Adaptive, portable, multi-sensory aid for the disabled

ABSTRACT

A mobile terminal is used to assist individuals with disabilities. The mobile terminal (e.g., a “smartphone” or other commercially available wireless handheld device) may be loaded with software. The software may be configured to: (i) receive information about a sensory deficit associated with a user, (ii) receive information about a sensory proficiency associated with the user, (iii) determine whether an event associated with a sensory deficit satisfies a criterion, and if so (iv) provide an assistive output based on a sensory proficiency and the event.

RELATED APPLICATIONS

The present application is a divisional application of U.S. patentapplication Ser. No. 12/704,119, filed Feb. 11, 2010, now abandoned.

The '119 application claimed the benefit of and priority to U.S.Provisional Patent Application Ser. No. 61/152,915 filed Feb. 16, 2009.

The contents of the applications listed above are hereby incorporated byreference in their entireties.

FIELD OF THE INVENTION

The present application is directed to assisting those with sensorydeficits through the use of a mobile terminal.

BACKGROUND

With a growing population, the number of developmentally disabledchildren grows. Additionally, the rates at which children have beendiagnosed as developmentally disabled, and particularly diagnosed withautism spectrum disorders (ASD), have steadily increased. Individualswith developmental disabilities often have several challenges in common,including but not limited to speech and language impairments, cognitivedeficits, social problems, behavioral problems, memory problems,attention deficits, and sensory processing dysfunction. Thedevelopmentally disabled population extends beyond those with ASD toinclude those with Down syndrome, cerebral palsy, Fragile X syndrome,and other disabilities which may involve sensory impairments. Inaddition, many individuals develop sensory deficits at later stages inlife, from trauma, old age, or other neuropathy. More than eight millionpeople over the age of five in the United States alone are hard ofhearing. More than seven million are visually impaired. Sensorydysfunction can result from stroke, brain injury, cancer, and otherailments.

For individuals with sensory processing dysfunction any number ofconditions such as: hypersensitivity or hyposensitivity with respect tosenses of sight; hearing; touch; smell; taste; pain (nociception);balance (equilibrioception), spatial orientation, joint motion andacceleration (proprioception and kinesthesia); time; and temperature(thermoception) may pose significant challenges in simple daily tasks,particularly when such tasks involve new environments. Many take forgranted the ability to visit a new place, walk in sunlight, cope withthe general auditory din of a public location, interact with a pet, andso on, though these tasks may be particularly daunting forhypersensitive individuals. For the hyposensitive, difficulty hearing,feeling, seeing, or balancing presents a constant battle, forcing manyto overcompensate for their deficits (e.g., aggressively bang an objectsimply to feel it), or exhaust themselves straining a particular sense.Many disabled individuals resort to self-stimulatory or “stereotypy”activity (e.g., echolalic or perseverative speech, repetitive bodymotions, etc.) as a mechanism to cope with or regulate sensoryirregularities.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a front elevational view of a mobile terminal.

FIG. 2 illustrates a block diagram of the components of a conventionalmobile terminal.

FIG. 3 illustrates a variety of networks in which a mobile terminal mayoperate.

FIG. 4 illustrates various devices which may interface with a mobileterminal.

FIG. 5 illustrates a first flow chart showing an exemplary embodiment ofthe present disclosure.

FIG. 6 illustrates a first screen shot associated with the flow chart ofFIG. 5.

FIG. 7 illustrates a second flow chart further explaining a sub-processwithin the process of FIG. 5.

FIG. 8 illustrates a second screen shot associated with the flow chartof FIG. 5.

FIG. 9 illustrates a third screen shot associated with the flow chart ofFIG. 5.

FIG. 10 illustrates a fourth screen shot associated with the flow chartof FIG. 5.

FIG. 11 illustrates a fifth screen shot associated with the flow chartof FIG. 5.

FIG. 12 illustrates a sixth screen shot associated with the flow chartof FIG. 5.

FIG. 13 illustrates a seventh screen shot associated with the flow chartof FIG. 5.

FIG. 14 illustrates an eighth screen shot associated with the flow chartof FIG. 5.

FIG. 15 illustrates a ninth screen shot associated with the flow chartof FIG. 5.

FIG. 16 illustrates a tenth screen shot associated with the flow chartof FIG. 5.

FIG. 17 illustrates an eleventh screen shot associated with the flowchart of FIG. 5.

FIG. 18 illustrates a twelfth screen shot associated with the flow chartof FIG. 5.

FIG. 19 illustrates a thirteenth screen shot associated with the flowchart of FIG. 5.

FIG. 20 illustrates a fourteenth screen shot associated with the flowchart of FIG. 5.

FIG. 21 illustrates a fifteenth screen shot associated with the flowchart of FIG. 5.

FIG. 22 illustrates a sixteenth screen shot associated with the flowchart of FIG. 5.

FIG. 23 illustrates a seventeenth screen shot associated with the flowchart of FIG. 5.

FIGS. 24A-C illustrate an eighteenth, nineteenth, and twentieth screenshot associated with the flow chart of FIG. 5, as well as a peripheraldevice which may interface with a mobile terminal, and an object whichemits a sound that may be detected by the peripheral device.

FIGS. 25A-C illustrate a twenty-first, twenty-second, and twenty-thirdscreen shot associated with the flow chart of FIG. 5, as well as anenvironment surrounding the mobile terminal.

FIGS. 26A-C illustrate a twenty-fourth, twenty-fifth, and twenty-xisthscreen shot associated with the flow chart of FIG. 5, as well asperipheral devices which may interface with a mobile terminal.

FIGS. 27A-C illustrate a twenty-seventh, twenty-eighth, and twenty-ninthscreen shot associated with the flow chart of FIG. 5, as well asperipheral devices which may interface with a mobile terminal, and anenvironment surrounding the mobile terminal.

FIG. 28 illustrates a first example database suitable for use with thepresent disclosure.

FIG. 29 illustrates a second example database suitable for use with thepresent disclosure.

DETAILED DESCRIPTION

Described herein are techniques for assisting individuals withdisabilities. In some embodiments, a mobile terminal (e.g., a“smartphone” or other commercially available wireless handheld devicedescribed herein) may be loaded with software of the present disclosure.The software may be configured to: (i) receive information about asensory deficit associated with a user, (ii) receive information about asensory proficiency associated with the user, (iii) determine whether anevent associated with a sensory deficit satisfies a criterion in that anevent has occurred that relates to or implicates the sensory deficit,and if so (iv) provide an assistive output based on a sensoryproficiency and the event. In one embodiment, a sensory input isreceived by the software and translated into a sensory output that ismore easily understood or processed by a user of a mobile terminal. Forexample, a user with auditory hypersensitivity may receive informationabout a loud environment (the presence of the loud environmentimplicates the auditory hypersensitivity) by way of a text shown on adisplay screen. In another example, a user with visual hypersensitivitymay receive information about an overly bright environment through audiospeakers or a peripheral headset. As an alternative to (or in additionto) such sensory translations, the software may provide therapeutic orinstructional output to help the disabled individual cope with the event(e.g., soothing music, a vibration, advice for how to cope with thesituation, a reassurance that a caregiver is nearby, or the like).

An overview of some of the hardware elements is provided beforeaddressing some of the methods and potential screen configurationsassociated with embodiments of the present disclosure. A mobile terminal10 is illustrated in FIG. 1. An exemplary suitable mobile terminal 10may be the APPLE® IPHONE™ (or IPOD™) and may include a user interface 12that includes a speaker 14, a microphone 26 (FIG. 2), a touch screendisplay 16, and a command entry button 18. One or more icons 20 may bedisplayed on the touch screen display 16. The user may enter commands bytouching an icon or element on the touch screen display 16 or by usingthe command entry button 18 as is well understood. One such icon 22 maylaunch software that effectuates embodiments of the present disclosure.While the IPHONE™ is particularly contemplated, the disclosure is not solimited, and any readily portable handheld computing device may beappropriate including personal digital assistants, cellular telephones,(e.g., so called “smartphones”), WINDOWS MOBILE™ enabled devices, or thelike. Further examples of such mobile terminals include, but are notlimited to: the BLACKBERRY™ line of wireless devices manufactured byResearch in Motion of Waterloo, Ontario, Calif.; the T-MOBILE G1™ orother mobile terminals compatible with the ANDROID operating system asdeveloped by Google, Inc. of Mountain View, Calif.; the PRADA™ or othermobile terminals manufactured by LG ELECTRONICS MOBILECOMM U.S.A, INC.of San Diego, Calif.; the INSTINCT™ or other mobile terminalsmanufactured by SAMSUNG TELECOMMUNICATIONS AMERICA, LLC; the XPERIA™ X1or other mobile terminals manufactured by SONY ERICSSON MOBILECOMMUNICATIONS AB of Sweden; the N96 or other mobile terminalsmanufactured by NOKIA GROUP of Finland; and the PRE™ or other mobileterminals manufactured by PALM, INC. of Sunnyvale, Calif. In otherembodiments, a mobile terminal may comprise a portable personal computerwith a sufficiently large, touch-sensitive screen (e.g., a “tablet” or“tablet computer” such as the APPLE® IPAD™).

A block diagram of the mobile terminal 10 is illustrated in FIG. 2. Themobile terminal 10 includes the elements already illustrated and furtherincludes, within the user interface 12, a camera 24 and theaforementioned microphone 26. The user interface 12 is operativelycoupled to a central processor (CPU) 28. The CPU 28 is also operativelycoupled to memory 30 with software 32 stored therein, an accelerometer34, transceiver circuitry 36, which in turn is coupled to an antenna 38,one or more wired ports 40, a battery 42, and any other miscellaneousperipherals 44 as is common in the mobile terminal art (e.g., aninternal clock, a vibration unit, a Global Positioning System (GPS)receiver, a thermometer, a light sensor, a proximity sensor, analtimeter, a pressure sensor or the like). To the extent that suchelements may interact with the user, they may also be considered part ofthe user interface even if not explicitly labeled such. For example,when a vibration unit vibrates, the user interacts with it by feelingthe vibration. Likewise, the output of the GPS receiver may be perceivedby the user through the display 16, through an output of the speaker 14,or the like. Any such miscellaneous peripherals 44 may alternately oradditionally be embodied as separate peripheral devices in communicationwith mobile terminal 10 (rather than as a component thereof). The CPU 28as operated under the instructions of the software is a control systemas that term is defined in the Rules of Interpretation & GeneralDefinitions set forth below. It should be understood that the hardwareconfiguration depicted by FIG. 2 is illustrative and that mobileterminal 10 may comprise other types of hardware, either as componentsof the mobile terminal 10, or as peripheral devices in communicationwith the mobile terminal 10. Such separate peripheral devices arediscussed further below with respect to FIG. 4.

FIG. 3 illustrates a variety of networks through which mobile terminals10 may communicate. For example, a first mobile terminal 10A maycommunicate with a second mobile terminal 10B through a local wirelessnetwork/protocol such as a BLUETOOTH™, infrared, or other short rangecommunication protocol. Additionally, or alternatively, the secondmobile terminal 10B may communicate with a cellular network such as thePublic Land Mobile Network (PLMN) 46 through a cellular base station 48.The PLMN 46 may communicate with the Public Switched Telephone Network(PSTN) 50 and/or the Internet 52. One or more computers 54 may becommunicatively coupled to the Internet 52 and may include a wirelesstransceiver to communicate wirelessly with a third mobile terminal 10Cor a docking station to communicate via a wired line with a fourthmobile terminal 10D. More information on networks and communicationprotocols can be found in the Rules of Interpretation and GeneralDefinitions set forth below. Suffice to say that the particular networkor communication protocol is not central to embodiments of the presentdisclosure. Additionally, it should be understood that such protocolsmay enable communications between mobile terminals and variousperipheral devices, including sensors, cameras, microphones, and thelike.

FIG. 4 illustrates an assortment of such peripheral devices 58 a-f.While other devices may communicate with the mobile terminal 10 asdescribed in this disclosure, a few are specifically shown. Anapplication programming interface (API) may enable interaction betweensoftware of the present disclosure and such separate devices.Communication methods which may be utilized between the peripheraldevices and the mobile terminal 10 include wireless protocols referencedabove (e.g., BLUETOOTH™, infrared), radio frequency identification(RFID), WiFi, and hard wiring.

Peripheral eyeglasses 58 a may communicate with the mobile terminal 10,and may comprise an optical camera 60, a RFID transponder (not shown),an audio microphone (not shown), accelerometer (not shown), and/or aninfrared transmitter (not shown). Eyeglasses 58 a may be used to provideinformation related to (i) the environment surrounding an individualwearing the eyeglasses (e.g., still images or video are taken fromoutward-facing optical camera 60 and transmitted to mobile terminal 10through eyeglasses 58 a), (ii) pupil dilation of the individual wearingthe glasses (e.g., optical-camera 60 is oriented inward to directlymonitor the user's eyes), or (iii) the direction of the individual'sgaze with respect to the mobile terminal (e.g., the orientation ofeyeglasses 58 a relative to mobile terminal 10 is determined using RFID,and thus the orientation of the individual's gaze relative to mobileterminal 10 may be inferred). As an example of eye-tracking technologyused in conjunction with a mobile terminal, the 3D Eyetracking UI asdeveloped by TAT THE ASTONISHING TRIBE, AB of Sweden for the T-MOBILE G1mobile terminal alters the user-interface presented by the mobileterminal's display screen based on the orientation of the device to theuser's eyes.

Peripheral skin patch 58 b may comprise (i) a sensor enabling themeasurement of biological properties, and (ii) a transmitter enablingwireless communication with mobile terminal 10. Biological propertiesmeasured by skin patch 58 b may include temperature, blood glucose (theGUARDIAN® REAL-Time Continuous Glucose Monitoring System, as produced byMEDTRONIC MINIMED, INC. of Northridge, Calif., comprises a sensorembedded within a skin patch that monitors blood sugar levels and atransmitter to send data wirelessly to a receiver via BLUETOOTH™), bloodpressure, Galvanic Skin Response (GSR) or electrodermal skin response,heart rate, and cardiac electrical activity (e.g., a wireless ECG patchsuch as the one developed by the IMEC research center in Belgium). Thus,a user of mobile terminal 10 may also wear skin patch 58 b such thatbiological measurements may be taken in a continuous, periodic, orotherwise specified manner.

Wristwatch 58 c may measure and transmit biological data in a mannersimilar to that of skin patch 58 b. Thus, a user of mobile terminal 10who enjoys wearing a watch may at the same time provide a variety ofbiological data used by the software of the present disclosure. Avariety of commercially available digital wristwatches comprise sensorsfor measuring such data (e.g., the IRONMAN® Heart Rate Monitor Watch byTIMEX®). In an alternate embodiment, wristwatch 58 c may instead takethe form of a basic wristband, arm band or ankle band (i.e., comprisinga sensor and transmitter but no clock or stopwatch functionality).Likewise, while not shown, the wristwatch 58 c may also include anaccelerometer, an RFID transponder, or the like to detect movement ofthe wristwatch 58 c or its position relative to the mobile terminal 10.

Ring 58 d may also measure and transmit biological data and/orpositional data in a manner similar to that of skin patch 58 b. The ring58 d and its components (e.g., sensors, electronics) may be fashionedfrom a variety of metals, plastics, composites, and/or semiconductors.In one embodiment, a commercially available ring, such as the MYBEATHeart Rate Ring by LIFESPAN FITNESS, may be adapted for use withsoftware of the present disclosure.

In one embodiment (not shown), a biosensor may be permanently affixed toor implanted beneath the user's skin. For example, a subcutaneouselectrochemical sensor may be used to continuously measure and report onblood (or interstitial fluid) glucose level. An example of a deviceemploying such a semi-permanent sensor is the FREESTYLE NAVIGATOR®Continuous Glucose Monitoring System by ABBOTT LABORATORIES of AbbottPark, Ill.

Clip-on microphone 58 e may be used to receive audio data andre-transmit it to the mobile terminal 10. Thus, if the clip-onmicrophone 58 e is oriented toward the user's mouth (as it appears inFIG. 4), the user of mobile terminal 10 may act and speak naturally,with the user's speech transmitted automatically to mobile terminal 10.In another embodiment, the clip-on microphone 58 e may be oriented awayfrom the user such that sounds occurring in front of the user may becaptured (e.g., fastened to a lapel as in FIG. 4, but facing outward;fastened to a belt clip, purse, or other area; etc.). In one embodiment(not shown), a microphone and speaker may both exist as part of awireless headset worn by a user of mobile terminal 10 (e.g., affixed viaan ergonomic piece of plastic so as to rest around a user's ear).Suffice it to say that wireless microphones of a variety of sorts arewidely available in the commercial marketplace, well known in the mobileterminal art, and may be used in conjunction with software of thepresent invention.

Headphones 58 f may be employed as a mechanism to hear audio produced bymobile terminal 10 (e.g., as an alternate to, or in addition to, speaker14). In one embodiment, headphones may take the form of earbuds, andalong with ergonomic plastic 64, may be positioned in a manner that issecure, comfortable, and effective for listening to sounds produce bythe mobile terminal 10. In other embodiments, a full set ofear-encompassing headphones may be used. As alluded to above, headphones58 f may also be incorporated into a single device with a microphone 58e (not shown). Suffice it to say that headphones of a variety of sortsare widely available in the commercial marketplace and may be used inconjunction with software of the present invention. In some embodiments,such headphones may incorporate other sensors (e.g., RFID, an infraredsensor, a motion sensor) to facilitate embodiments of the presentdisclosure (e.g., determination of the physical orientation of theuser's head).

Worth discussing at this point is the manner in which mobile terminal 10may be carried or held. In various embodiments, a mobile terminal 10 maybe carried or held by a disabled individual and/or a physically presentcaregiver. At times, mobile terminal 10 may be carried by hand, but thismay not always be desirable, convenient, or possible. While the abovedescribed peripheral devices 58 may allow for the placement of mobileterminal 10 inside of a pocket or a purse, some embodiments (e.g., thoserequiring substantially continuous use of components integrated withinmobile terminal 10 itself) may benefit from the placement of mobileterminal 10 outside of a pocket or a purse. In one embodiment, asdepicted in FIG. 4, a lanyard 62 may be attached to mobile terminal 10(or, attached to an after-market case designed to protect the mobileterminal 10 from wear and tear), such that mobile terminal 10 may hangfrom the neck of an individual, rather than be held by hand, within apocket or within in a purse. Other cases (not shown) that fasten mobileterminal 10 to the body or clothing may be used, including arm bands,wrist bands, belt clips, and the like. Still further cases (also notshown) may affix mobile terminal 10 to wheelchair arms, head pointers,or other physical devices used to afford the disabled with mobility orthe ability to communicate. Peripheral devices 58 may be similarlyaffixed to such locations or objects.

In some embodiments, a user of the mobile terminal 10 may be reminded orencouraged to appropriately position the mobile terminal 10 or anyassociated peripheral devices on or around his or her person. Forexample, in some embodiments, placement of the mobile terminal 10 aroundthe neck of a disabled user through the use of lanyard 62 may bedesirable. Thus, if the mobile terminal's integrated light sensordetects lack of light for a prolonged period of time during a specifiedcoverage period (e.g., suggesting that the device may be in the user'spocket or purse), a vibration unit, speaker 14, or other output devicemay be actuated to alert the user and encourage him or her toappropriately position the mobile terminal 10 (e.g., the mobile terminalvibrates, and the speaker outputs a ringtone or other alarm). Anaccelerometer 34 and/or light sensor may then detect that the device hasbeen picked up, at which point instructions for effectively positioningthe device may be output (e.g., the display screen 16 outputs agraphical indication of the device hung around a neck, while speaker 14outputs an audio instruction to “Hang around your neck, please!”).Placement of the mobile terminal 10 or associated peripheral devices inother locations may be encouraged in a similar manner (e.g., a user isencouraged to position peripheral microphone 58 e outward such that itcaptures the speech of those engaged in conversation with the user).

Against this backdrop of hardware, an overview of an exemplary method ispresented starting with reference to FIG. 5 as a flow diagraminterspersed with additional Figures for supporting explanatory andexemplary screen shots. Initially, a user acquires a mobile terminal 10(block 100). This acquisition may be a gift, a purchase, a lease,borrowing, or otherwise. For example, the disabled individual maypurchase the mobile terminal 10 and loan it to the caregiver for thepurposes of carrying out embodiments of the present disclosure.Alternatively, the caregiver may purchase the mobile terminal 10 for useby and/or with the disabled individual. The caregiver may load thesoftware embodying the present disclosure onto the mobile terminal 10(block 102). If the mobile terminal 10 has a disk drive or other datastorage device or functionality (e.g. flash memory), the software may beloaded from a disk or other source. Otherwise, the software may beloaded onto a computer 54 (e.g., via disk transfer or download) andtransferred to the mobile terminal 10 through the docking station orwirelessly. As still another alternative, the mobile terminal 10 maydownload the software through the PLMN 46 or other technique as desired.The software could be preloaded on the mobile terminal 10 such that whenthe lease or purchase is made, the software is immediately accessible.The software itself may be free, paid or subsidized in whole or part bya third party (such as a school district or other governmental agency orprivate foundation). Note that while it is contemplated that thecaregiver performs these and other steps, it is also possible that thedisabled individual performs these and the other steps.

The caregiver may then configure elements of the software, so as tocustomize the software on behalf of a disabled individual. Particularly,the caregiver may provide the software with information about one ormore sensory deficits associated with a user, and thus the softwarereceives information about a sensory deficit associated with the user(block 104). For example, one or more particular hypersensitivities orhyposensitivities associated with a disabled user may be described. Thesoftware may further receive information about a sensory proficiencyassociated with the user (block 106). Further, the caregiver mayprovide, and the software may store, a criterion for providing anassistive output as well as the actual assistive output (block 108). Theprovided information is saved, and the software, through the sensorsassociated with the mobile terminal 10 monitors the activities of thedisabled individual to determine if the criterion has been satisfied(block 110). If the criterion has been satisfied, the mobile terminal10, or an associated peripheral device 44 or 58 may be used to providean assistive output. Many of these steps will be further explained belowwith reference to various exemplary screen shots.

In an exemplary embodiment, after the software is loaded onto the mobileterminal (block 104, FIG. 5), the caregiver accesses the software, suchas by pressing the icon 22. Once the software has launched, thecaregiver may be presented with a screenshot such as that illustrated inFIG. 6 (or one with comparable functionality). From this screen, thecaregiver may configure the software as further described below.Alternatively, the caregiver may configure the software through anothermechanism such as by selecting a “Settings” icon from a group of icons20 displayed by the mobile terminal 10. As yet another alternative,computer 54 may be used to configure software settings, which may thenbe transmitted to a mobile terminal 10 using methodologies describedabove (e.g., a caregiver uses Web-based software on a personal computerto provide sensory deficit information, such that the information issaved to a central database and subsequently accessible wirelessly bythe mobile terminal 10; a caregiver uses a software program of apersonal computer to provide sensory deficit information, and theinformation is subsequently transferred from computer 54 to mobileterminal 10 via docking station).

Regardless of the particular technique, the caregiver may see a screensuch as that illustrated in FIG. 6. Specifically, as shown, thecaregiver may be presented with a “Profiles” command and a “GeneralSettings” command. In block 150 of FIG. 7, the user selects “Profiles”,in which case the user may see a screen such as that shown in FIG. 8 (orscreen with comparable functionality), where the user may select fromprofiles 200 that the user has previously created or from defaultpre-existing templates 202, or create a new template by activating the“+” command 204 (e.g., add new profile) in the upper corner. The“General Settings” may allow the user to configure certain defaultparameters that are not implicated by other rules within the software(e.g., if the user has auditory neutral, a default volume may be set; ifthe user is visual neutral, a default brightness may be set; or thelike).

If the user selects a pre-existing profile or template (block 152), theuser may be shown a screen like that shown in FIG. 9 (or screen withcomparable functionality), where the user may see the profile name 206,a particular sensory deficit defined for the profile 208, a particularsensory proficiency defined for the profile 210, and a particularassistive output defined for the profile 212. In the upper corner, an“Edit” command 214 may be present so that the user can enter the profileand change one or more parameters associated with the profile.Pre-existing templates may be profiles that are created as a startingpoint for serving those with particular sensory challenges (e.g., fieldspre-populated with information that may be applicable to a user withthermoception dysfunction, to a user with tinnitus and hyperacusis, orto a child user with Sensory Processing Disorder (SPD)).

If the user is satisfied with the current parameters associated with theprofile (i.e., elements 206, 208, 210, and 212), the user may select theload command 216 (block 154) in which case the software loads theprofile and begins monitoring to see if the criterion is satisfied asdescribed with reference to block 110 of FIG. 5. Alternatively, the usermay select the edit command 214 (block 156) in which case the processgoes to block 160 of FIG. 7.

If, instead of loading an existing profile, the user selects the “+”command 204, a blank profile is created (block 158) and the user may seea screen like that in FIG. 10 (or screen with comparable functionality).It should be noted that the edit command 214 and the add new profilecommand 204 may both take the user to a screen like that illustrated inFIG. 10. If the user is creating a profile, the parameters are blank.However, if the user is editing a pre-existing profile, then theparameters may be pre-populated with information that was previouslystored for that profile.

As shown, the screen of FIG. 10 allows for creation of a new profile(and, as noted above, editing of an existing profile is substantiallysimilar, albeit with pre-populated fields). Assuming creation of a newprofile, the caregiver may input and save various types of configurationdata or parameters in association with the profile, though some softwareconfigurations may be pre-programmed or set to “default” values beforethe caregiver begins this step (i.e., default settings are programmed bythe software's manufacturers (this may be particularly true of thetemplates)). Multiple profiles may be created for one or more disabledindividuals (e.g., separate profile names of “Jimmy's Daytime SensoryAmbassador”, “Jimmy's Nighttime Sensory Ambassador”, and/or “Sally'sSensory Ambassador”), allowing for use of the mobile terminal 10 bydifferent individuals at different times, or the same individual withrespect to different profiles, if so desired. The caregiver may utilizea keyboard feature (either hard-wired keys or a virtual keyboard, aswith the IPHONE™) to enter such a profile name (block 160). For eachprofile, the caregiver may also (as illustrated in FIG. 5): (i) add oredit information related to a sensory deficit (block 104) (i.e., for newprofiles, sensory deficit information may only be added, whereas forexisting profiles, sensory deficit information may be added or edited),(ii) add or edit information related to a sensory proficiency (block106) (i.e., for new profiles, sensory proficiency information may onlybe added, whereas for existing profiles, sensory proficiency informationmay be added or edited), (iii) add or edit assistive outputs (block 108)(e.g., translatory, therapeutic or informative outputs triggered byevents detected by the mobile terminal as described herein), and (iv)save a profile.

Continuing with the new profile example, and assuming the caregiver hasselected the “Add/Edit Sensory Deficit Info” option (FIG. 10), thecaregiver may be shown a screen such as that of FIG. 11 (or screen withcomparable functionality). The screen of FIG. 11 allows creators of newprofiles to choose a manner in which sensory deficit information will beprovided. Among the options are: (i) “Take a survey” (e.g., allowingusers to answer survey questions as they pertain to the sensorycapacities of a particular disabled individual), (ii) “Enter specificdeficit info” (e.g., allowing users to provide data or metrics about aspecific sensory deficit associated with a user), (iii) “Perform asensory evaluation” (e.g., whereby tests are performed by the software,in conjunction with sensors of the mobile terminal 10 and/or peripheraldevices, to determine sensory deficit information in association with aparticular user), and (iv) “Import data from another source” (e.g.,allowing users to upload or transfer electronic files containingpreexisting sensory deficiency data).

Continuing the example, and assuming the caregiver has selected the“Take a survey” option, a screen such as that of FIG. 12 (or a screenwith comparable functionality) may be presented. As shown by FIG. 12,for example, a caregiver may complete an “intake” survey orquestionnaire about a child with SPD (i.e., a neurological disabilitycausing difficulties with processing information from the senses),providing information about the child's specific sensory deficits. Sucha survey is known in the art as a “Sensory Profile,” “Sensory ProcessingDisorder Survey,” or “Sensory Processing Measure Survey,” and elementsof a prior art survey developed by Winnie Dunn, Ph.D. of the Universityof Kansas Medical Center may be used for purposes of this disclosure.When completing the survey, caregivers may mark “Always,” “Frequently,”“Occasionally,” “Seldom,” “Never” in response to various behaviorsassociated with a child's tactile sensitivity (e.g., “Expresses distressduring grooming”), taste/smell sensitivity (e.g., “Limits self tocertain food textures/temperatures”), movement sensitivity (e.g.,“Becomes anxious or distressed when feet leave the ground”),sensory-seeking behavior (e.g., “Enjoys strange noises/seeks to makenoise for noise's sake”), auditory filtering (e.g., “Is distracted orhas trouble when there is lots of noise around”), kinetic energy (e.g.,“Has a weak grasp”), visual/auditory sensitivity (e.g., “Covers eyes orsquints to protect eyes from light”), etc. When responses to the surveyare complete, the caregiver may select a “Save Survey” or similarlylabeled option (not shown). As will be described later herein, in someembodiments, survey responses may be used by the software to create orsuggest one or more assistive outputs (e.g., a user who always expressesdistress during grooming and frequently limits himself to certaintextures/temperatures may be over-sensitive to tactile stimulation, andso an assistive output may be provided to the user if a tactilestimulation is detected).

Returning to FIG. 11 and instead assuming the caregiver selects the“Enter specific deficit info” option, one or more screens such as thatshown by FIG. 13 (or screen with comparable functionality) may beoutput. The caregiver may utilize such screens to provide data relatedto sensory deficits known to be experienced by a particular user of themobile terminal 10. Such data may be provided using one or more of avariety of different measurement techniques, including: (i) binarymeasurement (e.g., a user either has an impairment, or does not; forexample, a user is hypersensitive to touch and vestibularlyhyposensitive), (ii) interval or scaled measurement (e.g., informationrelated to a deficit is entered using a scale such as “1 through 10”,“low/medium/high”, “always/frequently/occasionally/seldom/never”,“severe/mild”, “1 through 100%”), (iii) ordinal or ranked measurement(e.g., impairments are ranked in relation to one another, such that auser's tactile hypersensitivity is ranked as a primary impairment, withvestibular hyposensitivity as a secondary impairment, and auditoryhypersensitivity as a tertiary impairment; one individual's impairmentsmay be ranked in relation to another individual's, such that a child'sauditory filtering capability is in the 37th percentile among allchildren, or a child ranks “below average” with respect to motionsensitivity), and/or (iv) provision of a specific numeric levelassociated with a sensory deficit (e.g., “20/80” is entered to describevision, a frequency range and decibel level are entered to describehearing loss). For demonstrative purposes, the screen of FIG. 13 shows amixture of measurement types appearing on a single screen, though inpractice, the process for entering specific deficit info may implicatenumerous screens, with any or all of the above measurement typesutilized in association with any deficit. When the process of enteringspecific deficit information is complete, the caregiver may select a“Save Deficit Information” or similarly labeled option (not shown).

Returning to FIG. 11 and instead assuming the caregiver selects the“Perform a sensory evaluation” option, one or more screens such as thatshown by FIG. 14 (or screen with comparable functionality) may beoutput, allowing a particular user's sensory capacities to be tested orevaluated directly by the software of the present disclosure. Forexample, using a screen similar to that of FIG. 14, a user's hearing maybe evaluated. As is known in the art, a behavioral audiogram may becreated, whereby a user's hearing level in association with differentfrequencies may be measured by the software. For example, tones atdifferent frequencies may each be output at different decibel levels(from quiet to loud), and the user may indicate whether or not he or shecan hear the tones as they increase in volume (touch-screen options 66 aand 66 b may allow for affirmative responses related to the left andright ears, respectively). The result of such a process is the creationof an audiogram for the user, which spells out precisely the deficits(or even proficiencies and hypersensitivities) experienced by the userin terms of hearing (e.g., presbyacusis, or the downward-sloping loss ofhearing at progressively higher frequencies, as is common with the humanaging process). As described below, such data may be used by thesoftware to create or suggest one or more assistive outputs (e.g.,translation of a sound detected at high-frequency, such as a bird call,to a picture output by display screen 16, such as a picture of the birdthat emitted the sound). Of course, other senses besides hearing may beso tested by the software. For example, a user's tactile sensitivity maybe tested using a series of outputs from a vibration unit (e.g.,progressively longer or more forceful vibrations are output, and theuser responds affirmatively when the vibrations can be felt by shakingthe mobile terminal 10 such that motion is recognized by theaccelerometer 34). A user's balance or equilibrioception may be testedusing the accelerometer 34 (e.g., an evaluation is administered wherebythe user must attach the mobile terminal 10 securely to a belt clip, andthen walk a balance beam, with shifts in balance measured by theaccelerometer 34 and reported to the software). A user's visual acuitymay be tested by (i) projecting a Snellen-scale visual acuity chart ontoa wall using a peripheral projection device associated with the mobileterminal 10 (the EXPO™ smartphone by LG ELECTRONICS is an example of acommercially available mobile terminal that employs a projector, in thiscase a peripheral DLP projection device made by TEXAS INSTRUMENTSCORPORATION), (ii) outputting audio instructions for the user to stand aparticular distance from the surface onto which the chart is projected,(iii) outputting audio instructions to read aloud the letters shown oneach line of the chart, and (iv) employing a microphone and speechrecognition technology to determine whether the user has successfullyread the letters of each line (thus determining a level of visual acuityin association with the user). Other sensory capacities may also betested by the software to the extent possible. Of course, fordemonstrative purposes, FIG. 14 shows one example screen for conductingone type of sensory evaluation, though in practice, a variety of screendesigns may be employed to accommodate evaluation of a variety ofsenses, and yet further screens may be designed to facilitate the user'snavigation to and from such screens (e.g., after selecting the “Performa sensory evaluation” option shown by the screen of FIG. 11, a user isthen presented with a menu of different evaluation types, from which onemay be selected).

Returning again to FIG. 11, a user may select an option to “Import datafrom another source”. Selection of such an option would allow theelectronic transfer of preexisting data regarding any sensory deficitsassociated with a user. In one example, an electronic file may beuploaded locally from the memory of the mobile terminal 10. In anotherexample, a file may be transmitted electronically from another device(e.g., a file is transferred wirelessly from another mobile terminal 10,a file is transferred from a computer 54 to a mobile terminal 10 using adocking station). In yet another example, a file is downloaded from theInternet (e.g., using email, using a Web browser). In this manner,existing spreadsheets, charts (e.g., audiograms), word-processeddocuments (e.g., Sensory Profiles) or the like may be utilized by thesoftware, rather than having users key-in data that is alreadydocumented in some form.

Worth describing at this juncture are some sensory deficit types forwhich information may be provided. Again, though FIG. 13 shows only afew types of deficits appearing on a single screen, in practice,information for numerous different deficit types may be entered usingany number of screens (e.g., a menu system shows categories of deficits,allowing users to drill down from a high level of categorization toprovide a more detailed level of information for a particular deficit;for example, a user first selects a “deficits” option, then selects a“hearing” category, then selects “tinnitus,” then “objective tinnitus”,then “pulsatile tinnitus,” and finally enters information about theseverity of the particular user's pulsatile tinnitus).

Some users may suffer from a visual impairment or may have difficultyseeing. In one example, a user has a loss of visual acuity (e.g.,difficulty seeing clearly, whether short-sightedness, near-sightedness,astigmatism, imbalance of acuity between eyes, or overall lack of acuityor blindness). In another example, a user has a loss of visual field(e.g., a reduction in the total area that can be seen without moving thehead or eyes, such as a loss of tunnel vision). In another example, auser suffers from a lack of ocular motor control (e.g., difficultyfixating on objects of particular types or objects in particularlocations). In another example, a user has a hypersensitivity to brightenvironments (e.g., shields eyes in sunlight or when standing influorescent light) or to bright colors. In another example, a user iscolorblind.

Some users may suffer from a hearing or auditory impairment. In oneembodiment, a user has a lost an ability (in part or in full) to hearcertain frequencies of sound. For example, a use with mild hearing losshears only sounds at 25 decibels and above, whereas a user with profoundhearing loss hears only sounds louder than 90 decibels. A person'shearing sensitivity in association with particular ranges of frequenciesmay be plotted on an audiogram, as is known in the art. In anotherembodiment, a user may suffer from tinnitus, or the perception of soundwithin the human ear without a corresponding external sound. Tinnitusmay be objective (e.g., so-called “pulsatile” tinnitus heard by others,such as a rumbling of blood flow emanating from a vascular source withinthe body), or subjective (e.g., ringing, buzzing, clicking, or beepingnot heard by others). Some users with tinnitus may also suffer hearingloss, and/or may be more sensititive to sounds emitted at certainfrequencies (e.g., a particular ringing tone is uncomfortable to hear).Some users may experience hyperacusis (i.e., oversensitivity to certaintypes of sounds or volume levels).

Some users may experience sensory deficits related to touch. Forexample, some users may exhibit tactile defensiveness (hypersensitivityto touch). Some users may avoid contact with certain types of texturesor objects, such as wool, sand, hair combs, cardboard, or light humantouch. In other examples, users may suffer from tactile anesthesia, orinability to register sensation through the skin. For example, a usermay have lost, partially or completely, the inability to feel objectsusing a particular finger or hand.

Some users may experience impairments related to taste (gustatorysensations, as mediated by taste buds). For example, a user mayexperience ageusia, or the inability to detect sweetness, sourness,bitterness, saltiness in association with foods or beverages. Otherusers may experience hypogeusia (partial loss of gustatory sensation) ordysgeusia (the distortion or alteration of taste).

Some users may have difficulty smelling or accurately processingolfactory sensations. For example, a user with anosmia may not be ableto detect an odor. A user with hyposmia may experience a reduction inthe sense of smell (e.g., in association with a particular odor, or withall odors). Some may perceive odors which are not present (so-called“olfactory hallucinations”), or may be hypersensitive to certain typesof odors (e.g., the smell of metal or rubber makes a particular userfeel sick).

Some users may experience difficulty sensing temperature. For example,hyposensitive thermoception may result in the inability or impairedability to register or sense different temperatures (e.g., a user may“under-feel” heat, and so may be prone to kitchen or fireplace burns).Some users may be hypersensitive to temperature (e.g., moderate warmthis perceived as excessive heat; a somewhat cold object is perceived asextremely cold).

Some users may experience problems with balance or equilibrioception. Insome examples, balance-related challenges may stem from the vestibularsystem, resulting in impaired ability to sense body movement,accelerate, or maintain postural equilibrium. Further challenges relatedto equilibrioception may result in poor coordination, motor planning, orsequencing of actions.

Some users may suffer from challenges related to proprioception, or theability to interoceptively sense where body parts are located inrelation to one another. Some users may experience impairment related tothe sensation of joint position, resulting in difficulty determining thelocation of a body part in space. Some users may experience kinestheticimpairment, resulting in difficulty sensing that a certain body part hasmoved. Some users with proprioceptive and/or kinesthetic impairments mayexhibit clumsiness, insensitivity to pain, or sensory-seeking behaviors(e.g., walk, push objects, write, play with objects, or touch peoplewith excessive force).

Returning to the flowchart of FIG. 5, having already receivedinformation about any sensory deficits, the process continues to block106, wherein information related to sensory proficiencies are receivedin association with a user. For purposes of this disclosure, a sensoryproficiency will be considered a satisfactory level of functioningassociated with any of the senses described herein; in other words, if auser doesn't experience a deficit in association with a particularsensory capacity, such a sense may be considered fully functional orproficient.

Given this framework, perhaps the easiest way of receiving informationrelated to sensory proficiencies (block 106 of FIG. 5) may be to firstreceive information related to sensory deficits (block 104 of FIG. 5),and then infer that all sensory capacities which have not been expresslydescribed as deficient by a user to in fact be proficient. Such anoption is presented by the screen of FIG. 15, which allows a user toindicate (using a simple “ON/OFF” switch) that essentially all sensesnot described as impaired are indeed functioning appropriately. Thus,the user may first provide information related to sensory deficits in amanner similar to any or all of those described with respect to FIGS. 11through 14, and select this option, such that information related tosensory proficiencies may also be received (i.e., block 106 iscomplete).

Of course, in some situations, a caregiver or other user may wish toexpressly provide information related to one or more functional sensorycapacities (e.g., perhaps a particular sense associated with a user isexceptionally well developed, such as acute “20/10” vision). In such acase, a user may first set the switch shown by FIG. 15 to the “OFF”position, and then utilize any of the options shown, including “Take asurvey,” “Enter specific proficiency info,” “Perform a sensoryevaluation,” or “Import data from another source” (e.g., upon movementof the switch to the “OFF” position, the options are no longer “grayedout” or unavailable as shown by FIG. 15).

Once information related to sensory deficits and proficiencies arereceived by the software, the process depicted by FIG. 5 continues toblock 108, wherein the software stores a criterion for providing anassistive output. A criterion for providing an assistive output may alsobe thought of as an “assistive output rule,” or an instruction whichindicates that if an event associated with a sensory deficit isdetected, an assistive output based on a sensory proficiency should beprovided. Thus, the criteria described herein in essence may serve as“triggers” for the automatic provision of assistive outputs by thesoftware. Two overarching types of such criteria are contemplated: (i)those created automatically by the software in response to receivingsensory deficit and proficiency information at block 104 and block 106of FIG. 5, and (ii) those programmed explicitly by users of thesoftware. Each type will be discussed in turn.

In some embodiments, software of the present disclosure automaticallycreates assistive output rules based on sensory proficiency anddeficiency information provided by users. For example, sensoryproficiency and deficiency data associated with a user may be organizedinto a database (an example database is described below), and thedatabase may be scanned by the software. The purpose of the scan may beto identify trends within the data that would suggest certain types ofassistive outputs may be beneficial to the user. For example, programmedlogic within the software code may indicate that assistive output rulesare to be automatically created when certain fields within the databaseindicate certain data. For example, if one or more database recordsassociated with a user's visual acuity describe vision as proficient,and yet other records indicate the user suffers moderate to severehearing loss at high frequencies (4000 Hz and above), a rule mayautomatically be created whereby sounds (detected by integratedmicrophone 26 or clip-on microphone 58 e) occurring above 4000 Hz areautomatically “translated” into images and output via display screen 16(e.g., a chime from a clock is detected, and so a picture of a clock isoutput by the screen). Other examples of assistive output rules aredescribed below, but suffice it to say that any rule may be so createdby the software upon analyzing data provided by users; through time andexperience, creators of the software may learn that users withparticular deficits commonly program certain assistive output rules, andsuch information may feed the software's logic for automaticallycreating assistive output rules.

In other embodiments, assistive output rules may be created by users ofthe software. Turning to FIG. 10, such a process may begin when a userselects an “Add/Edit Assistive Outputs” option. The user may then bebrought to a screen similar to that shown by FIG. 16. An assortment ofassistive output rules is shown. In this example, we assume some ruleshave been created automatically by the software and others have beenprogrammed by a user. The rules include: “Translate high-frequency audioto pics” (e.g., created automatically by the software to translatehigh-frequency sounds to images), “Put sunglasses on when it's toobright” (e.g., created by a caregiver to help manage a user'shypersensitivity to light), “Rule 1 (11/15/10 4:57 pm)” (e.g., a ruleset by a caregiver but not expressly labeled, and so automatically givena label based on the time and date at which it was created), “Calmingaudio for nervous arm-flapping” (e.g., created by a caregiver to outputsoothing music and instructions via speaker 14 or headphones 58 f if auser's arms are flapping repeatedly as detected by an accelerometer inring 58 d or in the mobile terminal 10; a disabled individual whose armsare flapping may be engaged in proprioceptive self-stimulatory behavioras a coping mechanism for sensory overstimulation of some kind), “Rule 2(11/23/10 6:18 am)” (e.g., again, not labeled by a caregiver), “Convertpictures to touch-screen Braille” (e.g., an object detected byintegrated camera 24 or peripheral camera 60 is identified using anonline image-comparison database communicatively coupled to the softwarevia an API, such that a text-based description of the object may beoutput by display screen 16 as a sequence of vibrations representingBraille characters), and “Balance therapy after falling down” (e.g.,created automatically by the software such that if a user with knownequilibrioceptive challenges falls down (as detected by accelerometer34), a balance-related game may be output via display screen 16, as atool to help the user stand back up). Some of these example assistiveoutput rules will be described in more detail below. It is possible thatthe screen of FIG. 16 may instead show (i) no rules (i.e., none have yetbeen created) or (ii) categories of rules (i.e., such that users mayorganize rules by type). To continue the example, it is assumed that theuser selects the “+” option shown at the top-right of the screen, so asto create a new rule.

FIG. 17 illustrates a first example screen for configuring a newassistive output rule, in which the user is given the option to labelthe rule, such as by typing a name in the space provided. If the userchooses not to exercise this option and simply presses “Create,” atimestamp type label may be supplied as described above.

Once the user selects “Create,” a second screen for configuring anassistive output rule may be presented, an example of which is shown byFIG. 18 (or one of comparable functionality). Using a screen such asthis, the user may first choose a broad type of criterion (“trigger”) tobe further considered (at block 110 of the flowchart of FIG. 5, asdescribed below) in determining whether an assistive output should beprovided. Several categories of criteria are presented (though othersmay be used): “Sounds, noises or speech”; “Physical movement”; “Objectsor people near this device”; “Temperature”; “Lighting”; “The location ofthis device” and “A specific time or date”. These categories areexamples; other categorizations, subcategories, and navigationaltechniques (e.g., “searching” instead of browsing) may be employed tohelp a user specify a type of assistive output rule he or she would liketo create.

For example, assuming the user selects the option labeled “Sounds,noises or speech,” a screen such as that of FIG. 19 (or one ofcomparable functionality) may be presented, where various subcategoriesmay help the user refine the description of a type of detected audiothat should be used to trigger an assistive output. As shown,subcategories include “The user's speech,” “Another person's speech,”and “Common sounds or noises”. Continuing the example, it is assumed theuser selects “Common sounds and noises,” resulting in the output of ascreen such as that shown by FIG. 20 (or one with comparablefunctionality).

Using a screen such as this, the caregiver may begin to configure acriterion such that an identified sound (i.e., as detected by integratedmicrophone 26 or clip-on microphone 58 e) will be used to trigger anassistive output. Using one option, a user may search or browse analready-existing “Library” to select a particular sound to be used as atrigger. Using another option, the user may choose to record and save acustom sound.

If the caregiver selects the latter option, a screen such as that ofFIG. 21 (or one with comparable functionality) may be shown, enablingthe user to enter a name for the new sound (e.g., “tea kettle”). Afterthe user selects the “Save” option shown by FIG. 21, a screen such asthat of FIG. 22 (or one with comparable functionality) may be output.

The screen of FIG. 22 allows a user to record and save a customizedsound (in this example, the sound emitted by a particular tea kettleonce water boils). Controls 68 allow the user to Rewind, Stop, Play,Record, Skip (to a particular point within a recording), Undo and Save.Audio may be recorded by an integrated microphone 26 of the mobileterminal 10, by a clip-on microphone 58 e, or by another microphone incommunication with the mobile terminal 10 or computer 54. Once a soundis saved (control 68 a), an electronic file (i.e., digital audiowaveform) may be added to the software's internal library (such that thesound can be later discovered using the search and browse mechanismsshown in FIG. 20). Sound recognition functionality as it pertains tocomputer 54 and mobile terminal 10 is well known in the art, withsophistication and accuracy improving over time. For example, U.S. Pat.No. 6,990,445 to Ky describes a method of identifying a detected audiowaveform using a database of prerecorded audio waveforms; this documentis hereby incorporated by reference for all purposes. WINDOWS MEDIA®PLAYER as developed by MICROSOFT CORPORATION employs a form of audiowaveform matching termed “advanced audio fingerprinting,” to identifydigital audio files by comparing them within a database of known files.In another example, using tools readily available in the APPLE® IPHONE™software development kit, NUANCE COMMUNICATIONS, INC have developed asoftware application for the IPHONE called DRAGON DICTATION. Theapplication allows users of the IPHONE™ to dictate common speech; thespeech is then recognized so as to produce (i) text (e.g., saved to anotepad) or (ii) a command to execute a particular software function.Similar techniques may be utilized to detect sounds and speech forpurposes of the present disclosure.

Continuing with the example and assuming the user has selected the“Save” control 68 a, the user has now specified a criterion (i.e., thedetection of a sound produced by a tea kettle) for the provision of anassistive output. To complete programming the assistive output rule, theuser may now define the assistive output that should be produced shouldthe criterion be satisfied. To do so, one or more screens such as thatof FIG. 23 (or one with comparable functionality) may be utilized.

The user may then select one or more types of assistive outputs,including but not limited to: (i) visual outputs, (ii) audio outputs,and (iii) tactile outputs (e.g., a vibration). As shown by FIG. 23, theuser has selected “Visual output,” which may allow (e.g., through one ormore subsequent screens) the user to configure various media whichshould be presented by the display screen 16, including: (i) text (e.g.,as shown, a user can select an option to add text, and type any desiredtext into a provided field), (ii) one or more static images (e.g., asingle picture output for a period of time, two pictures outputsimultaneously, three pictures output in sequence, etc.), (iii) one ormore videos (e.g., as shown, a user can select an existing video, suchas one stored in the memory of the mobile terminal 10, or record a newvideo using an integrated or peripheral video camera), (iv) combinationsof the above (e.g., a picture with a text caption underneath instructingthe user to perform the action shown in the picture). It should beunderstood that a variety of configurations and preferences mayadditionally be specified by users at this time, even if they are notexplicitly illustrated by FIG. 23. For example, the user mayadditionally configure: (i) sequences for presenting different visualmedia as part of an assistive output (e.g., a second picture shouldappear 10 seconds after a first picture), (ii) the location or size ofpictures, text or video on a screen when an assistive output is produced(e.g., a picture should appear underneath a video); (iii) specificdisplay devices other than screen 16 which should be utilized inconjunction with the assistive output (e.g., a second display screen orperipheral projection device associated with the mobile terminal 10should alternately or additionally be utilized). Finally, the user canselect an option to “Save” any configured visual outputs, as shown.

If the user were alternately or additionally to select the “Audiooutput” option of FIG. 23, an assistive audio output may be configuredand saved. Configuration choices may include: (i) a Text-to-Speech (TTS)option, allowing a user to type text which will then be “read aloud” bya computerized voice when the assistive output is executed; (ii) anoption to load a saved audio file (e.g., a song file saved in a commonformat such as MP3, WAV or AAC; a voice recording; a sound effect; etc.)or record a new audio file (e.g., using an integrated or peripheralmicrophone; and (iii) an option to adjust the volume associated with anaudio output. Further configuration options (not shown) may allow formultiple types of audio to be played simultaneously or in sequence,using one or more peripheral speakers or a peripheral headset, and soon.

If the user were alternately or additionally to select the “Tactileoutput” option of FIG. 23, an assistive vibratory or tactile output maybe configured and saved. Configuration choices may include: (i) anoption to output a standard vibration (e.g., using an integratedvibration unit of the mobile terminal 10), including sub-options toconfigure the length and intensity of the vibration (e.g., the userconfigures a “medium”-intensity vibration that is to last until a userpicks up the mobile terminal, as detected by a sensor); and (ii) anoption to output a sequence of vibrations. For example, a disabled enduser may be blind or have significantly impaired vision. A caregivermay, before transferring the mobile terminal 10 to the disabledindividual, select an option to output a sequence of vibrations ofdifferent intensities used to represent one or more conventional Braillecharacters. As an example of the representation of two-by-three Brailledot matrices on a touch-screen device, using the NOKIA 770 INTERNETTABLET, which features a piezoelectric material built into the touchscreen that vibrates when an electric signal is applied to it, thescientists at the University of Tampere in Finland created software thatrepresents a raised dot within a Braille matrix as a single pulse ofintense vibration, and an absent dot within a Braille matrix a longervibration made up of several weaker pulses. Other methodologies ofrepresenting Braille using a touch screen may be so employed. Further,other three-dimensional objects (e.g., words, pictures) may berepresented using a touch-screen associated with mobile terminal 10,using other techniques. For example, as described in U.S. PatentApplication No. 2009/0174673 to Ciesla, cavities associated withtouch-screen buttons may be inflated and deflated by a pump coupled to afluid reservoir, in subsets or individually, to providethree-dimensional tactile guidance to a user interacting with atouch-screen; this document is hereby incorporated by reference for allpurposes.

After configuring such preferences, the user selects the “Save AssistiveOutput Rule” option such that the new rule is created and added to thelist of saved rules shown by FIG. 16. A variety of rules catering to avariety of sensory deficits may be programmed in this manner.

With the rule now programmed, the process depicted by the flowchart ofFIG. 5 continues to decision block 110, wherein it is determined whetherthe criterion saved in association with the rule has been satisfied. Ifthe criterion is satisfied—e.g., a specified event has occurred, such asan utterance of a particular word or phrase, or increase in ambientlight as detected by a light sensor—the process continues to block 112,wherein the assistive output is provided. If the criterion is notsatisfied, the process loops at decision block 110 (notwithstandingother instructions saved in association with an assistive output rule,the software continually determines whether criteria are satisfied).

A variety of sensors, peripherals, peripheral devices, and othertechnologies associated with the mobile terminal 10 may be employed whendetermining whether a specified event has occurred. Some examples willnow be set forth in which particular technologies are paired with thedetection of particular behaviors, environmental changes or properties,occurrences, and the like. Though an example is not provided for everypossible pairing, this should not be construed as limiting the capacityof the software to employ all available technologies in detectingwhether criteria have been satisfied.

In some embodiments, a criterion may be satisfied if a sound, a noise orhuman speech is detected. Either or both of an integrated microphone 26and a peripheral microphone, such as clip-on microphone 56 e, may beemployed to help detect such audio occurrences. In one example ofdetecting speech, an integrated microphone 26 is used to detect that aparticular word or phrase has been spoken. In another example, a clip-onmicrophone 56 e is used to detect the word or phrase. In any case, anuttered word or phrase may be “matched” to a voiceprint (or waveform)stored within electronic memory. The voiceprint may be generic (e.g.,the speech criterion is satisfied the word or phrase may be spoken byany person), or associated with a particular person (e.g., the speechcriterion is satisfied only if the word or phrase is spoken by aparticular person). Various technologies for so-called biometricvoiceprint matching using mobile terminals are known in the art (e.g.,the PHONEFACTOR system manufactured by PHONEFACTOR, INC of OverlandPark, Kans. utilizes such technology). Of course, other sounds besidesspeech may be detected (e.g., human sounds such as crying or screaming,animal sounds such as a dog's bark or a bird's chirp, sounds occurringin an urban environment such as a car's horn, sounds occurring at homesuch as those emitted by home appliances). In one example, as described,an audio waveform detected by a microphone may be matched to aprerecorded waveform for purposes of identifying the detected waveform,using technologies similar to those incorporated by reference above. Insome embodiments, a database of sounds may be stored locally by themobile terminal 10. In other embodiments, a central server may store adatabase of sounds, and the mobile terminal 10 may communicate with sucha database for the purposes of identifying detected sounds. One suchdatabase is maintained by COMPARISONICS® CORPORATION of Grass Valley,Calif. Another such system, the SOLAR (Sound Object Localization andRetrieval) system as developed by researchers at the University ofIllinois at Champagne-Urbana, compares sounds detected by microphones toa vast database of sound types, so as to distinguish car horns, dogbarks, trains, human voices, etc. Software of the present disclosure maycommunicate with such a database using an API. In some embodiments, inorder for a criterion to be satisfied, noise must be above a thresholddecibel level (e.g., as specified by a user). In another embodiment, fora criterion to be satisfied, a sound must occur a certain number oftimes within a specified time period (e.g., a repeated sound suggests auser may be engaged in auditory sensory-seeking behavior).

In some embodiments, a criterion may be satisfied if physical movementis detected. In some embodiments, physical motion is associated with auser of the mobile terminal 10. For example, certain bodily motionsperformed by a user may be detected using peripheral devices 58 a-f. Inone example, an arm flapping motion is detected through the employ ofring 58 d, watch 58 c or other peripheral device attached to an arm.Waving of the hand, punching and hitting may be so detected. In oneembodiment, covering of the ears or eyes may be detected using acombination of sensors (e.g., eyeglasses 58 a and ring 58 d), with suchactions suggesting the user may be engaged in sensory-avoidant behavior(e.g., the environment is too bright or too loud for the user toeffectively process). In another example, a peripheral anklet maytransmit data to the mobile terminal 10 that suggests the user may berunning, kicking, or shaking his or her leg. In another example, rockingin place or shaking of the head are detected using eyeglasses 58 a(e.g., a motion sensor, infrared sensor, RFID transponder, or othersensor is embedded within the eyeglasses, communicating relative headposition to the mobile terminal 10). In one example, eyeglasses 58 e areused to determine that the user's gaze is oriented toward the mobileterminal (e.g., using camera 60, an RFID transponder, an infraredsensor, or the like). In another example, a separate peripheral deviceassociated with an object, person or area (e.g., an RFID receiver orother sensor placed in a particular location) may be used in conjunctionwith eyeglasses 58 e (e.g., worn by a disabled individual) to determinethat a disabled individual's gaze is indeed oriented in the direction ofthe object, person or area (e.g., for at least a threshold percentage oftime during a particular time period); for example, if a user's gaze isaffixed in a direction away from a source of light, sound, or otherstimulus, it may be determined that the user is engaged insensory-avoidant behavior.

Further, in some embodiments, any of the above motions may be detectedwithout the use of a separate peripheral device. For example, anintegrated camera 24 or motion sensor may detect motion activity (e.g.,as facilitated by the hanging of mobile terminal 10 around a user's neckusing lanyard 62, or by the attachment of the mobile terminal 10 to abelt clip, the sensor is oriented to detect activity motion activityoccurring in front of the user). In some embodiments, motion must lastfor a predetermined duration, or occur at a particular velocity, for acriterion to be satisfied. Accelerometer 34 and/or an integratedaltimeter may detect the mobile terminal 10 (and perhaps thus its user)has fallen to the ground. Also, direct physical interaction with themobile terminal 10 may be considered. For example, a pressure sensorassociated with a button or screen of the mobile terminal 10 may detectan excessive amount of force, an accelerometer 34 may detect excessivemotion, or a light sensor may detect a pattern indicative of tactilesensory-seeking behavior (e.g., the light sensor is repeatedly coveredand uncovered). In another example, a user repeatedly attempts to lowerthe volume associated with speaker 14 or a peripheral headset,suggesting auditory overstimulation. In some embodiments, combinationsof several of the above behaviors may lead the software to conclude thatcriteria have been satisfied. For example, simultaneous crying above athreshold decibel level and physical mishandling of the mobile terminal10 are illustrative of tactile sensory-seeking. In another example,tactile interaction with the mobile terminal 10 is first detected (e.g.,using a pressure sensor), and then accelerometer 34 determines thedevice has been dropped or put down, suggesting tactile avoidance.

In some embodiments, the detection of a nearby person or object maysatisfy a criterion. In one example, such a person or object may wear orcarry a device equipped with a sensor, and the sensor may in turncommunicate the presence of the object or person to the mobile terminal10. For example, one or more sensors or other technologies associatedwith a first mobile terminal 10 (e.g., carried by a disabled individual)may be used to determine that a second mobile terminal 10 or peripheraldevice (e.g., carried by another individual) is present. Manytechnologies may be employed to accomplish such a goal, including amotion sensor, infrared sensor, RFID, GPS, triangulation of geolocationusing WiFi or cellular network nodes. In one such embodiment, a firstand second mobile terminal 10 may be registered (e.g., using software ofthe present disclosure) as “buddy” or “friend” devices (e.g., tofacilitate the determination that a specific person or “friend” isnearby a disabled individual at a particular time). In anotherembodiment, a microphone may be used to detect a voiceprint associatedwith another person (e.g., a specific caregiver) or sound associatedwith a particular object.

In another embodiment, an optical camera, such as an integrated camera24, or a camera associated with a peripheral device, such as camera 60,may be used to detect the presence of a particular person or object.Either or both of a still camera and video camera may be so utilized,and such cameras may capture images periodically, continuously, upondemand of a user, or in response to instructions stored within thesoftware (e.g., an instruction states to actuate a camera in response toa loud spike in ambient volume as detected by a microphone). Forexample, image comparison search may be employed, allowing for imagescaptured by a camera to be matched with those included within adatabase. In one embodiment, the database is stored locally. Forexample, a user takes a digital photograph and stores it within memoryof the mobile terminal 10, such that a camera may capture a new imageand compare it to the stored digital photograph (e.g., if a match isdetected, a criterion is satisfied). In another embodiment, the databaseis maintained by a central server communicatively coupled with themobile terminal 10. An example of a large online database of images usedfor image-comparison purposes is the TINEYE REVERSE IMAGE SEARCH ENGINEcreated by IDEE, INC of Toronto, Canada, which includes a commercial APIallowing third party software applications (such as the software of thepresent disclosure) to communicate with the database. In this manner,captured images may be analyzed to detect particular objects or people,and such detection may satisfy a criterion related to an assistiveoutput rule. For example, the detection of a fire or stove may bebeneficial to a person with a thermoceptive deficit, the detection of adog may be beneficial to someone with a known tactile hypersensitivityto fur, the detection of a flower may be beneficial to someone with apoor sense of smell, or the detection of a particular type of food maybe beneficial to someone with an impaired gustatory sense. Again,effective positioning of the mobile terminal and/or its peripherals mayassist in such detections; for example, a peripheral camera 60 must beoriented outward from the user for objects in front of the user to bedetected.

In some embodiments, an optical camera associated with mobile terminal10 may be used to recognize text, and a criterion may be satisfied uponthe detection of text (whether any text, or a specific word or phrase).For example, Optical Character Recognition (OCR) technology may be usedto identify letters. Methods for using OCR technology are well known inthe mobile terminal art. For example, a user with a NOKIA phone (such asthe N95) running the SYMBIAN operating system may take advantage of theNOKIA MULTISCANNER OCR feature, pointing the phone's optical camera atphysical text, such that the text is recognized by the phone's softwareand output via the phone's display. Recognized text may then be saved,and the language library used to help identify words may be updated overthe air. Such technology may be employed to assist a user with a visiondeficit, who may encounter objects with physical text in his or hernatural environment (e.g., books, signs, etc.), but may be challenged orunable to read such text.

In some embodiments, the detection of a specific amount of light in theenvironment proximate to the mobile terminal 10 may satisfy a criterion.For example, a light sensor may detect a particular abundance or lack oflight during a period of time (e.g., indicating that the mobile terminalis in direct sunlight, or in a dark environment).

In some embodiments, the detection of a specific environmentaltemperature may satisfy a criterion. For example, a thermometer may beused to determine that the mobile terminal is in a particularly coldenvironment (e.g., beneath 40° F.). In another example, a directionaltemperature sensor may determine that a nearby object is at or above athreshold temperature (e.g., a nearby stove, fire, or other object isexcessively hot). In another example, a Really Simple Syndication (RSS)feed may be monitored such that weather trends associated with themobile terminal's current environment are known.

In one embodiment, the detection of an altitude or altitude change maysatisfy a criterion. For example, an altimeter may be used to determinethat an altitude associated with the mobile terminal has dropped rapidlywithin a short period of time. In another embodiment an altimeter maydetect that the mobile terminal 10 is currently at a high altitude,which in and of itself may affect the sensory capacities of a user.

In one embodiment, the detection of a barometric pressure or change inbarometric pressure may satisfy a criterion. For example, a barometerdetects a change in atmospheric pressure, which satisfies a criterion.

In some embodiments, data concerning the geographic location of themobile terminal 10 may satisfy a criterion. As is known in the mobileterminal art, a current geographic location may be determined using GPStechnology, triangulation using WiFi or cellular network nodes, or thelike. In one example, a criterion may be satisfied if the mobileterminal 10 is within proximity of a particular geographic location at aparticular time. In another example, a criterion may be satisfied if themobile terminal 10 has strayed outside of a geographic “safe zone”expressly programmed by a user (e.g., a 10 mile radius surroundingschool and home). In another example, a criterion is satisfied if themobile terminal 10 has been detected within sufficient range of threeseparate geographic areas within a particular time (e.g., a childcarrying the mobile terminal has gone from school, to an after schoolprogram, and finally to home on a particular afternoon). In anotherexample, a criterion is satisfied if the mobile terminal 10 has beentaken to a “new” geographic location as per a database storingpreviously visited locations. In another embodiment, using GPS, WiFitriangulation, and/or cellular triangulation, the current location of amobile terminal 10 is determined to be in an area characterized byparticular sensory properties (e.g., a bus station is loud and dark; alibrary is quiet and bright; an amusement park implicates vestibularstimulation, a beach is hot, wet, bright and sandy; etc.).

In some embodiments, a property or state associated with a user's body(e.g., a biometric reading), or a change associated therewith, maysatisfy a criterion. For example, a change in a user's heart rate orother property may suggest the user is nervous or excited, which inconjunction with another occurrence (e.g., a loud noise), may suggestthat the user is struggling with sensory regulation. In someembodiments, a criterion is satisfied based on heart rate dataassociated with a user of the mobile terminal 10. For example, a user'sheart rate may be monitored through the employ of a patch 58 b,wristwatch 58 c, ring 58 d, or other wearable device that may detect andtransmit such data (e.g., a bracelet, anklet, wristband, etc.). Acriterion may be satisfied by a specified increase in, decrease in, orconsistent level of beats per minute (e.g., a user's heart rate remainsbeneath 60 beats per minute for a period of time). In some embodiments,a criterion is satisfied based on blood pressure data associated with auser of the mobile terminal 10. For example, a user's blood pressure maybe monitored through the employ of a patch 58 b, wristwatch 58 c, ring58 d, or other wearable device, such as an armband. A criterion may besatisfied by a specified increase in, decrease in, or consistent levelof pressure (e.g., a user's blood pressure spikes above 130/80). In someembodiments, a criterion is satisfied based on body temperature dataassociated with a user of the mobile terminal 10. For example, a user'sbody temperature may be monitored using a peripheral device describedherein (e.g., a patch 58 b, wristwatch 58 c, or ring 58 d comprising aninfrared or heat sensor). A criterion may be satisfied by a specifiedincrease in, decrease in, or consistent body or skin temperature. Insome embodiments, a criterion is satisfied based on electrodermal orgalvanic skin response (GSR) data associated with a user of the mobileterminal 10. Such data are used in the art to monitor and infer humanemotions (e.g., fear, anger, anxiety, startle response, orientingresponse). GSR data may be read and transmit using a peripheral device,with the device enabled to measure electrical resistance between twopoints on the user's skin. Commercially available sensors such as theGSR 2™ as produced by THOUGHT TECHNOLOGY LTD of Plattsburgh, N.Y. may beadapted for purposes of this disclosure. A criterion may be satisfiedbased on an increase or decrease in electrodermal activity (e.g., asharp increase may suggest a user has grown fearful, anxious ordisoriented).

In some embodiments, a criterion may be satisfied based (in part or inwhole) on the occurrence of a particular time or date. For example acaregiver who knows that a user with SPD is scheduled to attendoccupational therapy on Wednesday at 4:30 p.m. might create an assistiveoutput in conjunction with this occasion. In one example, the assistiveoutput occurs before the occasion, to prepare the user mentally for theupcoming experience (e.g., a display screen reads, “Almost time for OT!Remember, one leg at a time when you're on the ropes!” and presents apicture of a rope ladder). In another example, a specific time frame isset (e.g., Wednesday between 4:30 and 4:40), such that if a criterion issatisfied during the specified time period, an assistive output shouldbe produced (e.g., if a camera detects a trampoline during the timeperiod, or an accelerometer detects a bouncing motion during the timerperiod, a previously recorded video of the user successfully jumping onthe trampoline is output via display 16, to model to the user anappropriate behavior when using the trampoline). In other words, in someembodiments, both (i) a criterion related to input from a sensor and(ii) a time/date criterion must be satisfied for an assistive output tobe produced. An internal clock or calendar coupled to CPU 28 may assistwith the determination of a current time and/or date.

Having described a wide array of criteria that may trigger an assistiveoutput, and manners for detecting the satisfaction of such criteria, thediscussion now turns to block 112 of the flowchart of FIG. 5, wherein anassistive output is provided. As described previously with respect toFIG. 23, a user programming an assistive output selects one or moredesired output types (e.g., text and a vibration), and any contentassociated therewith (e.g., types in and saves a block of text to beoutput). Thus, the assistive output as configured by the user in block108 of FIG. 5 is now initiated in block 112.

Generally speaking, certain communication types of assistive outputs maymake particular sense when matched to certain sensory deficits andproficiencies received in blocks 104 and 106 of FIG. 5 (e.g., for a userwith a hearing deficit and visual proficiency, audio is converted totext; for a user with tactile hyposensitivity and auditory proficiency,sensory-seeking interaction with the mobile terminal involving excessiveforce may be discouraged with an audio instruction). Rather than walkthrough each possible arrangement, four particular examples will now bedescribed wherein a user programs criteria for an assistive output, thecriteria are satisfied, and the assistive output is produced asinstructed.

In the first example, illustrated by FIGS. 24A-C, a user with a hearingdeficit is prompted with a visual and tactile output upon detection ofan audio event. In this particular example, a user has saved anelectronic file capturing the audio that should be used to trigger theassistive output; as in FIG. 20, FIG. 24A shows a screen whereby theuser records and saves a noise associated with a tea kettle. In FIG.24B, clip-on microphone 58 e picks up the sound emitted by the teakettle 70, such that software of the present invention (operating onmobile terminal 10) detects a “match” between the detected sound and thepreviously recorded audio waveform. Of course, microphones other thanclip-on microphone 58 e may be so utilized (e.g., integrated microphone26, or another type of peripheral microphone communicatively coupled tothe mobile terminal 10). Turning to FIG. 24C, as a result of thedetection of the audio event, an assistive output is produced,comprising: (i) a visual output of screen 16, including a picture of thetea kettle 70 and text indicating “Tea is ready!” (i.e., as programmedusing the screen of FIG. 20), and (ii) the actuation of a vibration unitto stimulate the user's sense of touch (illustrated by motion marks 72).

In the second example, illustrated by FIGS. 25A-C, a disabled user withsensitivity to light is prompted with both visual and audio assistiveoutputs upon detection of a brightness level that exceeds a threshold.For example, a caregiver may first specify a maximum level of brightnessor brightness threshold (e.g., anything above “Normal Daylight” shouldtrigger an assistive output), and then use a screen such as that of FIG.25A to configure assistive outputs that should be produced upon thethreshold being crossed (e.g., two pictures, some text, and an audiovoice memo should be output). Turning to FIG. 25B, the disabled user maythen carry the mobile terminal 10 into direct sunlight 74, such that alight sensor 76 detects that the predefined brightness threshold hasbeen exceeded. (The screen of FIG. 25B announcing that the threshold hasbeen succeeded may be optional.) The assistive outputs are then providedas shown in FIG. 25C, with text and pictures visually suggesting to theuser to put on a pair of sunglasses (via display screen 16), along withan audio recommendation of the same (via integrated speaker 14, thoughalternately or additionally, a headset or headphones 58 f may beemployed).

In the third example, illustrated by FIGS. 26A-C, a caregiver configuresthe software to output calming audio when it is detected that a disableduser (e.g., one with Down syndrome, Fragile X or autism) has beenflapping his arms. In some cases, disabled individuals flap their armsor engage in other proprioceptive self-stimulatory behavior as a copingmechanism for sensory overstimulation. The caregiver in this examplebegins by setting trigger criteria as shown in FIG. 26A, whereby a levelof hand motion (rapid) and length of motion activity (15 seconds) arespecified. The mobile terminal 10 is then given to the disabledindividual, who carries the device, and indeed begins to engage inself-stimulatory behavior, flapping his arms repeatedly for a period oftime. As shown by FIG. 26B, using peripheral ring 58 d, the motionactivity is detected, though it should be understood that a variety ofother sensors or technologies described herein may alternately oradditionally be used in detecting such activity. FIG. 26B also depictsan example screen that may optionally be output at this time, includinga text notification that at least 15 seconds of rapid hand motions havebeen detected, and so accordingly, two audio files will be output insequence. The first, “Calmnote (1/26/10)”, is a voice memo recorded by acaregiver in hopes of soothing or calming the individual when played(e.g., the caregiver says, “Jimmy: are you nervous or excited? Listen toyour favorite song and take a deep breath”). The second, “Harvest Moon”,is a song enjoyed particularly by the disabled individual during calmermoments. Turning to FIG. 26C, the mobile terminal 10 plays the audiothrough headphones 58 f, though other output devices including speaker14 and other types of peripheral headphones or headsets may be used.

In the fourth example, illustrated by FIGS. 27A-C, a caregiverconfigures the software such that when a particular image is recognizedby an optical camera communicatively coupled to the mobile terminal 10,a translatory tactile output is produced that describes the imaged. Suchan arrangement may benefit a disabled user of the mobile terminal 10 whosuffers from profound vision loss or blindness. As shown by FIG. 27A, acaregiver may begin by identifying an image, such as one stored withinthe memory of the mobile terminal that should be used as a benchmark towhich a detected image may be compared. In this example, the caregiverhas identified an image called “EXIT.GIF,” which depicts a common EXITsign. The caregiver also supplies text to describe the image (e.g., thecaregiver uses a physical or virtual keyboard to type “EXIT”), and savesthe configuration. Turning to FIG. 27B, the mobile terminal 10 is givento a disabled user, who also dons a pair of glasses 58 a including anoptical camera 60 (i.e., the glasses are worn as a fashionable,outward-facing camera, and are not necessarily used to enhance vision).Sure enough, a detected image 78 of an EXIT sign above a doorway matchesa benchmark image 80 saved in memory of the device, causing a criterionto be satisfied and an assistive output to be produced. In this example,turning now to FIG. 27C, a tactile assistive output is produced, in theform of a pattern of raised portions of a touch screen 16. As shown, thepattern of raised portions of touch screen 16 represents the word “EXIT”in Braille. As described previously (with respect to U.S. PatentApplication No. 2009/0174673 to Ciesla), cavities associated withportions of a touch screen may be inflated and deflated (with liquid orwith air, administered by a pumping mechanism) to raise such portions inthree dimensions. In this example, inflated cavities 82 a areinterspersed with deflated cavities 82 b, creating four differentBraille letters (each two-by-three matrix is associated with oneletter). Thus, by running a finger across the touch screen 16, thedisabled user may understand that an “EXIT” sign has been detected. Ofcourse, alternate methods of representing words and pictures usingtactile outputs may be employed as described above (e.g., thosediscussed with respect to the software developed by University ofTampere in Finland, which instead represents Braille characters using aseries of vibrations of different intensities).

Of course, many other examples exist of sensory-translational,therapeutic and instructional outputs in response to a variety ofevents. In one example, an individual with poor thermoceptive capabilitycarries a mobile terminal, and a thermometer detects a sudden andextreme increase in heat; audio speaker 14 then indicates “It's hot inhere—please be careful” while a display screen 16 shows image of athermometer, and a map of the environment indicating the location of“hot spots”. In another example, a user suffers from a visualimpairment, resulting in an inability to read. An optical camera detectsa sign that says “Baseball Field Ahead”. Headphones 58 f may then outputinstructions routing the user to the baseball field (e.g., inconjunction with GPA technology). In another example, a caregiverspecifies that a disabled user of a mobile terminal 10 is hypersensitiveto certain types of sounds (e.g., sounds of crowds and heavy machinery).The mobile terminal 10 may then detect using GPS that it is approachinga geographic location known for such types of sounds (e.g., a shoppingmall, a busy downtown area, a construction zone), and output aninstruction to avoid the area. In another example, a sensor detectsproximity to a nearby object on behalf of a user who is hyposensitive tovisual-spatial sensation (depth perception), and alerts the individualto the presence of the object using audio. In another example, a usersuffers impairments to the vestibular system, resulting in poor balanceand motor planning. Over time, an accelerometer 34 and/or altimeterdetect that the user seems to frequently lose his or her balance aftereating. Accordingly, immediately after lunch every day, a “balance game”is output, focusing the user on balancing an object shown by the mobileterminal's display screen 16 (e.g., balance a bowling ball as it rollsdown an alley), honing vestibular regulation. In another example, a useris hyposensitive to touch, and commonly engages intactile-sensory-seeking behavior by banging or otherwise aggressivelymanipulating objects. A pressure sensor detects the user is clasping themobile terminal 10 and/or pressing buttons with unnecessary force, andso the mobile terminal 10 vibrates vigorously such that user receivesthe tactile sensory stimulation he or she may have been seeking. Inanother example, a user may suffer from a proprioceptive deficitresulting in frequent fits of disorientation. An increased heart rateand increased level of electrodermal activity are detected. In response,an image of a caregiver is output via the display screen 16, along witha pre-recorded voice recording in which the caregiver says “It's OK,take a deep breath and sit down for a minute”. In another example, auser has impairments related to both vision and equilibrioception.Whenever a high degree of motion is detected by a motion sensor, awarning is output for the user to proceed with caution.

The methods described above lend themselves to a database of saved userprofiles. Such a database may store software settings in associationwith particular users (e.g., assistive output rules in accordance with“Jimmy Smith”). One exemplary user profile database is depicted by FIG.28. As shown, each record of the user profile database may be given aunique identifier (e.g., P-0001). Each record may then store variousdata in association with the unique identifier. A profile name asentered by a caregiver may be saved (e.g., “Jimmy Smith”). An indicationof whether or not the profile is currently loaded for use by thesoftware may also be saved. Within each profile, various user-programmedsettings may be stored. For example for each assistive output rule thatis programmed, the database may store: (i) a first trigger (e.g., anindication of what should initiate an assistive output, such as adetected object near the mobile terminal 10); (ii) a first trigger type(e.g., lighting, physical movement, the presence of an object near amobile terminal 10); (iii) any sensors or other technologies implicatedin detecting the first trigger (e.g., a light sensor, a peripheral ring56 d, a motion sensor, an optical camera); (iv) a second trigger, secondtrigger type, and sensors implicated in detecting the second trigger (ifapplicable; not shown); (v) a date/time associated with the trigger, ifapplicable (e.g., the rule is “in effect” on Wednesday from 4:30 to 4:40p.m.); (vi) one or more output types (e.g., visual, audio, tactile); and(vii) any content or instructions associated with an output (e.g., Rule1 implicates the output of text “Hi Jimmy, Is it too bright? Then put onyour sunglasses on” along with the output of two pictures,“jimmyoutside.JPG” and “Sunglasses.GIF”). Thus, as software of thepresent disclosure is programmed, data may be stored, and subsequentlyaccessed to facilitate the execution of process steps shown by theflowchart of FIG. 5. If necessary, such a database may include otherfields not shown by FIG. 28 (e.g., to indicate any preferences or otherdata associated with a rule, such as volume settings for an audiooutput, or a sequences in which images should be output). The databasemay be stored locally by mobile terminal 10, or by a central server incommunication with the mobile terminal 10.

The methods described above may also lend themselves to a database ofsaved sensory deficiency and/or proficiency data associated with a user(i.e., a “sensory profile database”). Turning to FIG. 29, each record ofthe sensory profile database may be given a unique identifier (e.g.,P-0001). Each record may then store various data in association with theunique identifier. A profile name as entered by a caregiver may be saved(e.g., “Jimmy Smith”). An indication of whether or not the profile iscurrently loaded for use by the software may also be saved. Within eachprofile, various data related to the sensory deficits and proficienciesof the user may be stored. For example, as users complete the processesof providing such data (whether by taking a survey, entering specificinformation related to a deficit, conducting a sensory evaluation, orimporting data from another source), as described above with respect toblocks 104 and 106 of the flowchart of FIG. 5, the data may be stored insuch a database. For demonstrative purposes, a limited number of fieldsfor such data are illustrated by FIG. 29, though it should be understoodthat such a database may contain quite a bit more data in practice, andalso may be organized differently. Nonetheless, the database of FIG. 29stores: (i) answers to questions from a SPD survey (e.g., as shown inFIG. 10, the user “Always” expresses distress during grooming,“Frequently” limits self to certain textures/temperatures, and “Never”becomes anxious or distressed when feet leave the ground), (ii) anindication that the user has an impaired sense of smell, (iii) anindication that the user does not experience hyperacusis(oversensitivity to sounds), (iv) indications that the user suffers frommoderate pulsatile tinnitus, (v) data indicating the user has “20/30”vision, (vi) indications that the user has severe tactile defensivenessto sand, and (vii) the results of a behavioral audiogram (e.g.,demonstrating the lowest decibel level at which the user can hearcertain frequencies of sounds in each ear).

In some embodiments, one or more fields of such a database may beupdated based on the user's interaction with a mobile terminal. Forexample, initially, a user may hear sounds at 2000 Hz only when theyoccur at or above a level of 40 dB (e.g., database fields for “LEFT EAR@ 2000 Hz” and “RIGHT EAR @ 2000 Hz may both store an indication of “40dB,” based on a first sensory evaluation conducted when the user beganusing the software). However, over time, the user's level of hearing maydecline. Such a decline may be detected (i) in response to normal use ofthe mobile terminal (e.g., as sounds are periodically output at afrequency of 2000 Hz, the software determines that the user is less andless responsive to them), and/or (ii) as a result of a second sensoryevaluation (e.g., one month after the user performs a first hearingtest, a second hearing test is conducted, producing a new behavioralaudiogram, such that the sensory profile database may be updated withnew information on the user's hearing). For example, if the user'sability to hear sounds emitted at a frequency of 2000 Hz has decreasedto the point where only sounds above a level of 50 dB are heard, thesensory profile database may be so updated. Of course, other sensorydeficit and proficiency information besides hearing information may beso updated (e.g., a user is originally thought to be severely tactiledefensive, but over time does not demonstrate dissatisfaction inresponse to the mobile terminal's vibrations (the user does not drop thedevice or express an emotional outburst), and so the user's tactiledefensiveness may be downgraded from severe).

In other embodiments, one or more fields of a sensory profile databasebased on data specifically entered by a user related to a deficit. Forexample, a user may take a new survey, or otherwise enter or import newinformation.

In any case, as the data in the sensory profile database of FIG. 29evolve over time, so may the assistive rules stored within the userprofile database of FIG. 28. For example, as described above, thesoftware may contain logic for automatically creating (or editingexisting) rules based on patterns or correlations of sensory profiledata. For example, logic of the software may dictate that if a user'slevel of hearing drops from 40 dB to 50 dB at 2000 Hz, not only shouldan image be output, but a tactile vibration should also be produced(e.g., a field for “OUTPUT CONTENT 2” within the database of FIG. 28 isautomatically changed from “N/A” to “MED. VIBRATE; UNTIL PICKED UP”).Thus, in some embodiments, the database of FIG. 28 may automaticallyreact to changes within the database of FIG. 29.

While the above discussion provides a particularly contemplated set ofembodiments, the present disclosure is not so limited. Numerousalternate embodiments are also possible and within the scope of thepresent disclosure. These alternate embodiments are not necessarilymutually exclusive with one another or with the embodiments set forthabove. Rather, components of the various embodiments may be mixed andmatched as needed or desired. In one alternate embodiment, when a userconfigures an assistive output (e.g., as shown by the screen of FIG.23), the user may select an option to import a preexisting visual oraudiovisual aid created using a separate software program. For example,the IPROMPTS® application as developed by HANDHOLD ADAPTIVE® of Shelton,Conn., allows users of the APPLE™ IPHONE™ and IPOD™ TOUCH to create andpresent visual picture schedules, visual countdown timers, and visualchoice boards to those with language and behavioral challenges (many ofwhom also suffer from sensory challenges). Such a preexisting scheduleincorporating pictures and text may be imported within the software ofthe present disclosure, and used as an assistive output. Methods for thecreation and presentation of such visual aids are described inApplicants' co-pending U.S. patent application Ser. No. 12/391,871,filed Feb. 24, 2009 and entitled “Portable Prompting Aid for theDevelopmentally Disabled”; the entirety of this document is incorporatedby reference for all purposes.

In some embodiments, a log of assistive output activity may bemaintained. For example, each time an assistive output is triggered, anindication of the output may be stored in a database (not shown). Thedatabase may store: (i) a time/date when the assistive output wasgenerated, (ii) the trigger for the assistive output, (iii) the type ofassistive output, and (iv) information related to events that transpiredbefore or the assistive output was produced (e.g., the database storesan electronic audio or video file of the user reacting to the output, anindication of any buttons pressed on the mobile terminal 10 after theoutput was produced, the current temperature or level of lighting at thetime of the output, etc.). Thus, those not present at the time of theassistive output (e.g., caregivers) may review the context in which aparticular assistive outputs is produced, or may review the data fortrends (e.g., certain types of assistive outputs are less common duringthe past month, perhaps indicating one of the user's sensory deficits isimproving). For example, a child's log indicates he more frequentlycovers his ears when the source of the noise is not man-made (e.g.,tractors, busses, televisions), and seems to struggle with his balancemore in the morning than at night. In another example, a user is morecommonly bothered by bright environments when the environment is alsonoisy, or it is winter time. Such data may be accessible through theuser's mobile terminal 10, may be uploaded or transmit to another mobileterminal 10 or computer 54. For example, the message shown by the screenof FIG. 26B is passed to a caregiver's mobile terminal 10 as analternative or in addition to appearing on a disabled user's mobileterminal 10.

In some embodiments, a remote observant may monitor the environment of adisabled individual. For example, a video or still camera transmitsimages from the disabled individual's mobile terminal 10 to a separatecomputing device (e.g., another mobile terminal or a personal computer),such that a remote observant (e.g., caregiver, therapist, call centeremployee) can view the images being sent in substantially real time. Forexample, an audio feed is transmitted to a remote parent, who can“listen in” to her child's environment. In another example, data fromvarious sensors of the mobile terminal 10 may be uploaded to a centralmonitoring Web site allowing those with access to view currentenvironmental data associated with a disabled individual.

Rules of Interpretation & General Definitions

Numerous embodiments are described in this disclosure, and are presentedfor illustrative purposes only. The described embodiments are not, andare not intended to be, limiting in any sense. The presently disclosedinvention(s) are widely applicable to numerous embodiments, as isreadily apparent from the disclosure. One of ordinary skill in the artwill recognize that the disclosed invention(s) may be practiced withvarious modifications and alterations, such as structural, logical,software, and electrical modifications. Although particular features ofthe disclosed invention(s) may be described with reference to one ormore particular embodiments and/or drawings, it should be understoodthat such features are not limited to usage in the one or moreparticular embodiments or drawings with reference to which they aredescribed, unless expressly specified otherwise.

The present disclosure is neither a literal description of allembodiments nor a listing of features of the invention that must bepresent in all embodiments.

Neither the Title (set forth at the beginning of the first page of thisdisclosure) nor the Abstract (set forth at the end of this disclosure)is to be taken as limiting in any way as the scope of the disclosedinvention(s).

The term “product” means any machine, manufacture and/or composition ofmatter as contemplated by 35 U.S.C. §101, unless expressly specifiedotherwise.

The terms “an embodiment”, “embodiment”, “embodiments”, “theembodiment”, “the embodiments”, “one or more embodiments”, “someembodiments”, “one embodiment” and the like mean “one or more (but notall) disclosed embodiments”, unless expressly specified otherwise.

The terms “the invention” and “the present invention” and the like mean“one or more embodiments of the present invention.”

A reference to “another embodiment” in describing an embodiment does notimply that the referenced embodiment is mutually exclusive with anotherembodiment (e.g., an embodiment described before the referencedembodiment), unless expressly specified otherwise.

The terms “including”, “comprising” and variations thereof mean“including but not limited to”, unless expressly specified otherwise.

The terms “a”, “an” and “the” mean “one or more”, unless expresslyspecified otherwise.

The term “plurality” means “two or more”, unless expressly specifiedotherwise.

The term “herein” means “in the present disclosure, including anythingwhich may be incorporated by reference”, unless expressly specifiedotherwise.

The phrase “at least one of”, when such phrase modifies a plurality ofthings (such as an enumerated list of things) means any combination ofone or more of those things, unless expressly specified otherwise. Forexample, the phrase at least one of a widget, a car and a wheel meanseither (i) a widget, (ii) a car, (iii) a wheel, (iv) a widget and a car,(v) a widget and a wheel, (vi) a car and a wheel, or (vii) a widget, acar and a wheel.

The phrase “based on” does not mean “based only on”, unless expresslyspecified otherwise. In other words, the phrase “based on” describesboth “based only on” and “based at least on”.

Where a limitation of a first claim would cover one of a feature as wellas more than one of a feature (e.g., a limitation such as “at least onewidget” covers one widget as well as more than one widget), and where ina second claim that depends on the first claim, the second claim uses adefinite article “the” to refer to the limitation (e.g., “the widget”),this does not imply that the first claim covers only one of the feature,and this does not imply that the second claim covers only one of thefeature (e.g., “the widget” can cover both one widget and more than onewidget).

Each process (whether called a method, algorithm or otherwise)inherently includes one or more steps, and therefore all references to a“step” or “steps” of a process have an inherent antecedent basis in themere recitation of the term ‘process’ or a like term. Accordingly, anyreference in a claim to a ‘step’ or ‘steps’ of a process has sufficientantecedent basis.

When an ordinal number (such as “first”, “second”, “third” and so on) isused as an adjective before a term, that ordinal number is used (unlessexpressly specified otherwise) merely to indicate a particular feature,such as to distinguish that particular feature from another feature thatis described by the same term or by a similar term. For example, a“first widget” may be so named merely to distinguish it from, e.g., a“second widget”. Thus, the mere usage of the ordinal numbers “first” and“second” before the term “widget” does not indicate any otherrelationship between the two widgets, and likewise does not indicate anyother characteristics of either or both widgets. For example, the mereusage of the ordinal numbers “first” and “second” before the term“widget” (1) does not indicate that either widget comes before or afterany other in order or location; (2) does not indicate that either widgetoccurs or acts before or after any other in time; and (3) does notindicate that either widget ranks above or below any other, as inimportance or quality. In addition, the mere usage of ordinal numbersdoes not define a numerical limit to the features identified with theordinal numbers. For example, the mere usage of the ordinal numbers“first” and “second” before the term “widget” does not indicate thatthere must be no more than two widgets.

When a single device or article is described herein, more than onedevice or article (whether or not they cooperate) may alternatively beused in place of the single device or article that is described.Accordingly, the functionality that is described as being possessed by adevice may alternatively be possessed by more than one device or article(whether or not they cooperate).

Similarly, where more than one device or article is described herein(whether or not they cooperate), a single device or article mayalternatively be used in place of the more than one device or articlethat is described. For example, a plurality of computer-based devicesmay be substituted with a single computer-based device. Accordingly, thevarious functionality that is described as being possessed by more thanone device or article may alternatively be possessed by a single deviceor article.

The functionality and/or the features of a single device that isdescribed may be alternatively embodied by one or more other devicesthat are described but are not explicitly described as having suchfunctionality and/or features. Thus, other embodiments need not includethe described device itself, but rather can include the one or moreother devices which would, in those other embodiments, have suchfunctionality/features.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. On the contrary, such devices need only transmit to eachother as necessary or desirable, and may actually refrain fromexchanging data most of the time. For example, a machine incommunication with another machine via the Internet may not transmitdata to the other machine for weeks at a time. In addition, devices thatare in communication with each other may communicate directly orindirectly through one or more intermediaries.

A description of an embodiment with several components or features doesnot imply that all or even any of such components and/or features arerequired. On the contrary, a variety of optional components aredescribed to illustrate the wide variety of possible embodiments of thepresent disclosure. Unless otherwise specified explicitly, no componentand/or feature is essential or required.

Further, although process steps, algorithms or the like may be describedin a sequential order, such processes may be configured to work indifferent orders. In other words, any sequence or order of steps thatmay be explicitly described does not necessarily indicate a requirementthat the steps be performed in that order. The steps of processesdescribed herein may be performed in any order practical. Further, somesteps may be performed simultaneously despite being described or impliedas occurring non-simultaneously (e.g., because one step is describedafter the other step). Moreover, the illustration of a process by itsdepiction in a drawing does not imply that the illustrated process isexclusive of other variations and modifications thereto, does not implythat the illustrated process or any of its steps are necessary to theinvention, and does not imply that the illustrated process is preferred.

Although a process may be described as including a plurality of steps,that does not indicate that all or even any of the steps are essentialor required. Various other embodiments within the scope of the describedinvention(s) include other processes that omit some or all of thedescribed steps. Unless otherwise specified explicitly, no step isessential or required.

Although a product may be described as including a plurality ofcomponents, aspects, qualities, characteristics and/or features, thatdoes not indicate that all of the plurality are essential or required.Various other embodiments within the scope of the described invention(s)include other products that omit some or all of the described plurality.

An enumerated list of items (which may or may not be numbered) does notimply that any or all of the items are mutually exclusive, unlessexpressly specified otherwise. Likewise, an enumerated list of items(which may or may not be numbered) does not imply that any or all of theitems are comprehensive of any category, unless expressly specifiedotherwise. For example, the enumerated list “a computer, a laptop, aPDA” does not imply that any or all of the three items of that list aremutually exclusive and does not imply that any or all of the three itemsof that list are comprehensive of any category.

Headings of sections provided in this disclosure are for convenienceonly, and are not to be taken as limiting the disclosure in any way.

“Determining” something can be performed in a variety of manners andtherefore the term “determining” (and like terms) includes calculating,computing, deriving, looking up (e.g., in a table, database or datastructure), ascertaining, recognizing, and the like.

A “display” as that term is used herein is an area that conveysinformation to a viewer. The information may be dynamic, in which case,an LCD, LED, CRT, LDP, rear projection, front projection, or the likemay be used to form the display. The aspect ratio of the display may be4:3, 16:9, or the like. Furthermore, the resolution of the display maybe any appropriate resolution such as 480i, 480p, 720p, 1080i, 1080p orthe like. The format of information sent to the display may be anyappropriate format such as standard definition (SDTV), enhanceddefinition (EDTV), high definition (HD), or the like. The informationmay likewise be static, in which case, painted glass may be used to formthe display. Note that static information may be presented on a displaycapable of displaying dynamic information if desired.

The present disclosure frequently refers to a “control system”. Acontrol system, as that term is used herein, may be a computer processorcoupled with an operating system, device drivers, and appropriateprograms (collectively “software”) with instructions to provide thefunctionality described for the control system. The software is storedin an associated memory device (sometimes referred to as a computerreadable medium). While it is contemplated that an appropriatelyprogrammed general purpose computer or computing device may be used, itis also contemplated that hard-wired circuitry or custom hardware (e.g.,an application specific integrated circuit (ASIC)) may be used in placeof, or in combination with, software instructions for implementation ofthe processes of various embodiments. Thus, embodiments are not limitedto any specific combination of hardware and software.

A “processor” means any one or more microprocessors, CPU devices,computing devices, microcontrollers, digital signal processors, or likedevices. Exemplary processors are the INTEL PENTIUM or AMD ATHLONprocessors.

The term “computer-readable medium” refers to any medium thatparticipates in providing data (e.g., instructions) that may be read bya computer, a processor or a like device. Such a medium may take manyforms, including but not limited to, non-volatile media, volatile media,and transmission media. Non-volatile media include, for example, opticalor magnetic disks and other persistent memory. Volatile media includeDRAM, which typically constitutes the main memory. Transmission mediainclude coaxial cables, copper wire and fiber optics, including thewires that comprise a system bus coupled to the processor. Transmissionmedia may include or convey acoustic waves, light waves andelectromagnetic emissions, such as those generated during RF and IR datacommunications. Common forms of computer-readable media include, forexample, a floppy disk, a flexible disk, hard disk, magnetic tape, anyother magnetic medium, a CD-ROM, DVD, any other optical medium, punchcards, paper tape, any other physical medium with patterns of holes, aRAM, a PROM, an EPROM, a FLASH-EEPROM, a USB memory stick, a dongle, anyother memory chip or cartridge, a carrier wave, or any other medium fromwhich a computer can read.

Various forms of computer readable media may be involved in carryingsequences of instructions to a processor. For example, sequences ofinstruction (i) may be delivered from RAM to a processor, (ii) may becarried over a wireless transmission medium, and/or (iii) may beformatted according to numerous formats, standards or protocols. For amore exhaustive list of protocols, the term “network” is defined belowand includes many exemplary protocols that are also applicable here.

It will be readily apparent that the various methods and algorithmsdescribed herein may be implemented by a control system and/or theinstructions of the software may be designed to carry out the processesof the present disclosure.

Where databases are described, it will be understood by one of ordinaryskill in the art that (i) alternative database structures to thosedescribed may be readily employed, and (ii) other memory structuresbesides databases may be readily employed. Any illustrations ordescriptions of any sample databases presented herein are illustrativearrangements for stored representations of information. Any number ofother arrangements may be employed besides those suggested by, e.g.,tables illustrated in drawings or elsewhere. Similarly, any illustratedentries of the databases represent exemplary information only; one ofordinary skill in the art will understand that the number and content ofthe entries can be different from those described herein. Further,despite any depiction of the databases as tables, other formats(including relational databases, object-based models, hierarchicalelectronic file structures, and/or distributed databases) could be usedto store and manipulate the data types described herein. Likewise,object methods or behaviors of a database can be used to implementvarious processes, such as those described herein. In addition, thedatabases may, in a known manner, be stored locally or remotely from adevice that accesses data in such a database. Furthermore, while unifieddatabases may be contemplated, it is also possible that the databasesmay be distributed and/or duplicated amongst a variety of devices.

As used herein a “network” is an environment wherein one or morecomputing devices may communicate with one another. Such devices maycommunicate directly or indirectly, via a wired or wireless medium suchas the Internet, Local Area Network (LAN), Wide Area Network (WAN), orEthernet (or IEEE 802.3), Token Ring, or via any appropriatecommunications means or combination of communications means. Exemplaryprotocols include but are not limited to: BLUETOOTH™, TDMA, CDMA, GSM,EDGE, GPRS, WCDMA, AMPS, D-AMPS, IEEE 802.11 (WI-FI), IEEE 802.3,TCP/IP, or the like. Note that if video signals or large files are beingsent over the network, a broadband network may be used to alleviatedelays associated with the transfer of such large files, however, suchis not strictly required. Each of the devices is adapted to communicateon such a communication means. Any number and type of machines may be incommunication via the network. Where the network is the Internet,communications over the Internet may be through a website maintained bya computer on a remote server or over an online data network includingcommercial online service providers, bulletin board systems, and thelike. In yet other embodiments, the devices may communicate with oneanother over RF, cellular networks, cable TV, satellite links, and thelike. Where appropriate encryption or other security measures such aslogins and passwords may be provided to protect proprietary orconfidential information.

Communication among computers and devices may be encrypted to insureprivacy and prevent fraud in any of a variety of ways well known in theart. Appropriate cryptographic protocols for bolstering system securityare described in Schneier, APPLIED CRYPTOGRAPHY, PROTOCOLS, ALGORITHMS,AND SOURCE CODE IN C, John Wiley & Sons, Inc. 2d ed., 1996, which isincorporated by reference in its entirety.

The present disclosure provides, to one of ordinary skill in the art, anenabling description of several embodiments and/or inventions. Some ofthese embodiments and/or inventions may not be claimed in the presentdisclosure, but may nevertheless be claimed in one or more continuingapplications that claim the benefit of priority of the presentdisclosure.

What is claimed is:
 1. A mobile terminal comprising: a plurality ofsensors, wherein the plurality of sensors includes at least two sensorsfrom a group that includes sensors operable to detect light, sound,motion, temperature, geographic location, blood glucose level, bloodpressure, electrodermal skin response, or heart rate; a user interface,wherein the user interface includes at least one of a touch-sensitivedisplay screen, an audio speaker, and a vibration unit as a userinterface component; and a control system operatively coupled to theplurality of sensors and the user interface and adapted to: receiveprofile information associated with a user of the mobile terminal,wherein the profile information is stored electronically in a databaseand defines: a sensory disability associated with the user of the mobileterminal, wherein the sensory disability comprises at least one among avision, hearing, locomotion, balance, smell, and touch disability; asensory proficiency associated with the user of the mobile terminal,wherein the sensory proficiency comprises a vision, hearing, or touchproficiency; an assistive output rule, which associates the sensorydisability with at least one sensor from the plurality of sensors,associates the sensory proficiency with at least one user interfacecomponent, and associates the sensory disability with the sensoryproficiency, such that at least one user interface component is employedin a manner that compensates for the sensory disability; and an eventcomprising a threshold criterion associated with the at least one sensorfrom the plurality of sensors; compare a measurement from the at leastone sensor to the threshold criterion to determine if the event hasoccurred; and execute the assistive output rule by employing the atleast one user interface component so as to compensate for the sensorydeficit of the user such that the user perceives the event through asense associated with the sensory proficiency and does not have to relyon a sense associated with the sensory disability to perceive the event.2. The mobile terminal of claim 1 wherein the at least one sensorcomprises a sensor positioned within a housing of the mobile terminal.3. The mobile terminal of claim 1 wherein the at least one sensorcomprises a peripheral device positioned outside of a housing of themobile terminal.
 4. The mobile terminal of claim 3 wherein theperipheral device is selected from the group consisting of: eyeglasses,skin patch, watch, ring, microphone, headphones, and camera.
 5. Themobile terminal of claim 4 wherein the peripheral device comprises aplurality of elements within the group within a single housing.
 6. Themobile terminal of claim 1 wherein the control system is adapted toreceive the profile from a caregiver.
 7. The mobile terminal of claim 1wherein the control system is further adapted to use a sensor todetermine if the mobile terminal is positioned in a desiredconfiguration and output a reminder to position the mobile terminal inthe desired configuration.
 8. The mobile terminal of claim 1 wherein thecontrol system is further adapted to receive the profile information byevaluating survey responses.
 9. The mobile terminal of claim 1 whereinthe control system is further adapted to receive the profile informationby evaluating selections made from a prepopulated list.
 10. The mobileterminal of claim 1 wherein the control system is further adapted toreceive the profile information by evaluating results from a diagnosisreceived through direct testing.
 11. The mobile terminal of claim 1wherein the assistive output uses a combination of user interfacecomponents.
 12. The mobile terminal of claim 1 wherein the sensorydisability comprises a hearing deficit and the sensory proficiencycomprises a touch proficiency.
 13. The mobile terminal of claim 12wherein the hearing deficit comprises hypersensitivity orhyposensitivity to sound.
 14. The mobile terminal of claim 1 wherein thesensory disability comprises a vision deficit and the sensoryproficiency comprises a hearing proficiency.
 15. The mobile terminal ofclaim 14 wherein the vision deficit comprises hypersensitivity orhyposensitivity to light.
 16. The mobile terminal of claim 1 wherein thesensory disability comprises a balance deficit and the sensoryproficiency comprises a vision proficiency.
 17. The mobile terminal ofclaim 1 wherein the sensory disability comprises a touch deficit and thesensory proficiency comprises a hearing proficiency.